Uniform Complexity for Text Generation
Authors: Joseph Marvin Imperial
Harish Tayyar Madabushi
Venue: EMNLP 2023 Findings (2023)
Keywords: text complexity
natural language generation
evaluation
narrative generation
TL;DR: We show that LLMs, specifically GPT-2 models, regardless of being finetuned, cannot uniformly match the complexity levels of their generated texts with respect to the prompts.

Abstract: Large language models (LLMs) have shown promising results in a wide array of generative NLP tasks, such as summarization and machine translation. In the context of narrative generation, however, existing models still do not capture factors that contribute to producing consistent text. For instance, it is logical that a piece of text or a story should be uniformly readable throughout and that this form of complexity should be controllable. As such, if the complexity of an input text prompt is rated first-grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform linguistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity in humans and generative models. From our results, we find that models such as GPT-2 struggle to preserve the complexity of input prompts used in its generations, even if finetuned with professionally written texts.

Uniform Complexity for Text Generation
Joseph Marvin Imperia P.A
Harish Tayyar Madabushi
University of Bath, UK National University, Philippines jmr meri2200bahh.om ac. uk htm43@bath.co ac.
Abstract
Prompt
Flesch Ease: 66.9
Large language models (LLMs) have shown promising results in wide array of genera- tive NLP tasks, such as summarization and ma- chine translation. In the context of narrative generation. however, existing models still do not capture factors consistent text. For instance, it logical that a piece of text or a story should be uniformly readable throughout and that this form of com- plexity should be controllable. As such, if the complexity an input text prompt is rated first- grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, introduce Uniform Com- plexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform lin- guistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity humans and generative mod- els. From our results, we find that models such GPT-2 struggle to preserve the complexity input prompts used in generations, even if finetuned with professionally written texts!.
discover grand hall filled with legendary weapons like Mjolnir and Excalibur. Each generation so, warriors come to the hall to inherit weapon that they worthy enough to wield...
Readable by grades 8-9 learners, Range: 60-70.
contribute to producing
GPT-2 SMALL
The Valkyrie led you in silence from moment she abducted you from your otherwise scemingly normal existence. You call her the Valkyrie because you do not know what else you call sevenfoot tall Nordic woman...
Continuation
Flesch Ease: 61.9
Figure I: An illustrated ideal example of generative model (c.g. GPT-2) producing uniformly complex text where the Flesch Ease score of the generated text falls within the same readability range the prompt which is 60-70 or 8th-9th grade reading level.
1 Introduction
Consistency is key in all aspects of the story- writing process. A story has to have consistent plot or story arc, observe a consistent use of fac- tual information needed, make use of consistent personas for characters, and follow a consistent writing style in order to be comprehensible (Guan and Huang, 2020; Zhang et al., 2022; Huang et al., 2023). While topic consistency is often used general metric for evaluating the quality of texts produced by narrative generation systems (Roem- mele, 2021), an important aspect that has received less attention from the rescarch community is the
investigation of consistency of readability or com- plexity of texts.
When crafting story, very reasonable for writer to ensure that all sentences are readable by a specific target audience (i.e., first-grade learners). As such, one must take note of common words within the general vocabulary of readers belonging to specific grade to avoid possible frustration in reading that will hinder effective comprehension (Gickling and Armstrong, 1978; Guevarra, 2011) Thus, text's complexity (and so the readability) depends largely on the writer's 's capability to ensure that the complexity of words and sentences con- sistent (Fountas and Pinnell, 1999; DuBay, 2004;
hhttps://githu. wttp://www..ommmmmaaaiiaiiiiccce uniform-complexin maifomm-comlexiity--eeegee

---

Agrawal and Carpuat, 2019). In the text generation process, since the oulput of generative models is controlled by the prompt, is essential that the complexity of the output to be consistent with the input prompt. Figure shows an ideal scenario where a prompt given by human has the same level of reading difficulty (66.9) as the generated text by GPT-2 model (61.9) based on the Flesch Ease formula (Flesch, 1948). The interpretation of Flesch Ease scores obtained means both texts can be read easily by Grade 8 and 9 learners.
160) used for evaluating text complexity in this process. We also report that fine-tuned models, even trained on uniformly complex and professionally written text, do not allevi- ate this problem.
2. Despite inconsistencies with respect to the prompts, we show that there are some similari- ties with the way GPT-2 models (fine-tuned or not) and humans generate continuations, such the preference of words from pre-defined word familiarity vocabularies such as Age-of- Acquisition and SubtlexUS.
Generally, the complexity of a narrative or any piece of text at hand can be measured by a multi- tude of content- based and linguistic factors. Some measures that have been explored through the years include syntactic complexity high-proficiency writing through the presence of qualified syntactic phrases or chunks such as words from a noun or verb phrase (Beers and Nagy, 2009;
3. Scoping our narrative more broadly, we sketch possible careful considerations when framing UCTG as an automatic evaluation metric for NLG systems due to the scalability of text complexity across language- tasks.
equivalence of
McNamara et al., 2010; Roemmele et al., 2017), discourse complexity by aggregating the presence of entities of a text such as mentions people, organi- zations, and locations which can lead to increased working memory burden (Feng et al., 2009), and vocabulary complexity by matching words from the text associated with a specific age-based diffi- culty level from developmental studies (Kuperman et al., 2012; Vajjala and Meurers, 2016) to name a few. These factors, when extracted from texts for evaluating complexity, usually follow a gen- eral linear property where the complexity of a text also increases as the value of measured feature is increased (Genzel and Charniak, 2002).
2 Benchmark Task Description
The proposed evaluation task can be generally adapted any NLG framework as long as texts are both the input and output. In the case of a narrative generation setting, given a series of prompts where each one is made up of a few sen- tences P = {pi, .e.............................. Pn and their continu- ations C = {e,, C,.,, by model M, linguistic feature function F processes instances from both the prompt and continuation sets get their corresponding measures F(px) E I and (c,,) E IR. All calculations performed by F are normalized? with either the number of word counts or a similar variable to avoid higher values from longer generated texts. A statistical test is then applied using the two groups to measure the sig- nificant differences in cach linguistic complexity feature variable.
In this study, aim to answer the following research question: can large language models (LLMs) generate responses that observe uni- form complexity with the prompt? To answer this, we introduce the novel task of Uniform Com- plexity for Text Generation (UCTG) with the aim of investigating responses obtained from any ar- bitrary generative language models used in story generation With GPT-2 models test case, we ex- plore how the outputs generated by LLMs relate to prompts and continuations given by humans, in relation to these linguistic features. Specifically, we make the following major contributions:
3 Generation Setup
We describe the training recipes and resources used for exploring UCTG in human and GPT-2 models.
3.1 Human Prompt and Continuation Data For the compilation of prompts and human con- tinuation benchmark, we used the WRITING- PROMPTS dataset collected by Fan al. (2018). This dataset derived from the r/WritingPrompts community of the Reddit platform where allows
1. We empirically show the inconsistencies in linguistic characteristics exhibited by LLMs, specifically GPT-2, with respect to the com- plexity of the prompts used. We cover wide array of linguistic complexity features (total
"This is a common practice in text complexity research to mitigate the effects length feature values (Lu, 2012). www.. reddit. weddit..co/rrreeeooo cww/r/w/rrc///ttee http///Writi tWritiiggrrmptss//////

---

users post story premises in various contexts and genres for other members to continue with their own ideas and creativity. The current compila- tion is divided into train, test, and validation splits. Given the nature of this task, we only used the test split which contains 15,138 pairs. In addition to the data preprocessing steps done by DeLucia et al. (2021), we handpicked prompts with at least 30 words in length to ensure that the neural models will have enough context for the generation phase while limiting human continuation texts to a mini- mum of 150 words and a maximum of 300 words to avoid fong-range memory limitation the small GPT-2 model used (Sun et al., 2021). Overall, we arrived at total of 941 prompt-human continuation pairs as the benchmark for the task.
the European Language Grid (ELG) compiled by Breuker (2022) which contains 1,200 curated stories in all levels of the Common European Framework of Reference for Languages (CEFR). We refer to both of these language resources as professionally-writter and uniformly complex as they have been produced and classified by expert writers in contrast to the WRITINGPROMPTS dataset in Fan et al. (2018) where the authors of narrative continuations are Reddit users and are non-experts. For the fine-tuning setup, we use the same recipe found in DeLucia et al. (2021). We upload the finetumed model in Hugginface for public access?.
For the decoding setup, DeLucia et al. (2021) reports that the best values for nucleus sampling top-p for the narrative generation task range from 0.70 to 0.90 wherein generated stories are more vivid and of better quality as evaluated through human and automatic means. Thus, for this work, we set top-p to 0.90 for all GPT-2 models.
3.2 Generation Models
To test whether large neural language models are capable of UCTG, we selected the GPT-2 class of models (Radford et al., 2019) for analysis due to its notable extensive use in the NLP community for the narrative generation task (See et al., 2019; Xu et al., 2020; Akoury et al., 2020; Chang et al.. 1)2 Aside from human generations hand, used three variations of GPT-24 for comparison as listed below:
Testing for (In)consistencies from the Prompt-Continuation Pairs
For the first experiment, test for possible sig- nificant differences in the complexity values of the prompts and generations of the three variations of GPT-2 models described previously plus gen- erations from humans. We use the Welch t-test (Welch, 1947) with Bonferroni correction resulting in new alpha level of a 0.0125 (0.05/4). We ex- tracted 160 linguistic features using the LingFeat tool by Lee et al. (2021) for the variables of interest. The tool covers the extraction of surface, lexical, phrasal, tree structure, type token, psycholinguis- tics, and formula-based readability features as fur- ther detailed in the subsections below. We only used the normalized or average-based linguistic complexity features (ex. count of noun POS / to- tal count of words) instead of total raw counts to avoid bias or reaching extremely high values for longer sentences. As reference for labels of mod- els used in the succeeding tables, we use GPT2 for the base model with no fine-tuning, GPT2-DF for the model fine-tuned for general narrative genera- tion, and GPT2-UF for the model fine-tuned with uniformly complex dataset.
Base GPT-2. This model is the standard off-the- shelf medium version with 355M parameters from Huggingface. We refer to this model as the base version as it is used without any fine-tuning.
Finetuned GPT-2 for General Narrative Gener- ation (DeLucia et al., 2021). This model uses the GPT-2 medium version that has been fine-tuned for general narrative generation using the same WRITINGPROMPTS dataset in Fan al. (2018) and has been optimized with the best hyperparam- eters for maximum fluency and diversity of content.
Finetuned GPT-2 with Uniformly Complex Texts. This model also uses the GPT-2 medium version but fine-tuned it using datasets from NEWSELA and EL... NEWSHLA is collection of 10,787 professionally-writenn news articles English with over 4-5 different variations in reading levels (Scarton et al., 2018; Xu et al., 2016, 2015). We also add another data source from
hntts::/Liveeerrgeea--aagggggg catalogue/corpus/ 1977
eu/ htttps://hugginge htt.aaaacccccoaaaaaeeaeaaaaa co/jjssphippeiial/ uniform-ccoplex- comtommmcoomle---t--mmme
"https://huggingfee gttpp "https://newsela" com/data/

---

For ease of readability and display purposes, we report the p-values from the result of the Welch test for the tables per linguistic group in the suc- ceeding sections. We also include an extensive set of barplot visualizations in Appendix A arranged alphabetically for ease of referencing.
Lase are integrated into most text-based applica- tions. From the results in Table 2, the majority of the narrative continuations from humans, base GPT-2 model, and both the finetuned GPT-2 mod- els were significantly different except for two. The formula for NARI is the only one that uses the num- ber of characters as a predictor, which may signal that lluaan-continee texts are more sensitive to character count. In addition, the base GPT-2 model obtaining con-sinnfficance for text dissimilarity on Flesch-Kincale may indicate potential preservation of complexity, However, this is a onc-off and is not consistent for all formulas explored.
4.1 Shallow Features
For our first feature set, we looked at 8 shallow or surface-based textual features as shown in Ta- ble 1 such as product and square root of total to- kens by total sentences, log densities of tokens to sentences, average token count per sentence, av- erage syllable count per sentence and per token, and average character count per sentence and to- ken. These linguistic features have been extensively used text complexity assessment in a wide range of languages such as in English (Flesch, I 1948), French (François and Miltsakaki, 2012) and Fil- ipino (Imperial and Ong, 2021). From the result, all of the mentioned features for the four groups appear to be significantly different with respect to the complexity of the prompt except for average characters per token for the finetuned GPT-2 model with uniformly complex data. From this, we posit that both humans and GPT-2 models, regardless of applied fine-tuning, generally ignore shallow or surface-basee characteristics of texts with respect to the prompt.
Feature
Human
GPT2
GPT2-DF
GPT2-UF
Flesch-Kincaid Kincuid
0.0008
0.0831
0.0000
0.0000
NARI
0,3835
0.0111
0.0000
0.0000
Coleman Limu
0.0000
0.0000
0.0000
0.0000
SMOG
0.0000
0,0000
0.00226
0,0000
Gunning-Fog Fog
0.0002
0.0000
Linsear
0.0000
0.0005
0,0000
0.0000
Table 2: Traditional formula-based features.
4.3 Part of Speech Features
We further the analysis by looking deeper into the text structure via part-of-speech (POS) tags. For this study, syntactic concepts covering nouns, verbs, adverbs, adjectives, subordinating clauses and con- junction, coordinating clauses and conjunction and prepositional phrases were used as predictors to calculate the densities of prompts and text contin- uations in both sentence and token level aspect (Heilman al., 2007; Lu, 2012). Table details 47 ratio and avcrage-based predictors, quantify-
Feature
Human
GPT2
GPT2-DF
GPT2-UF
Total token x Total sent
0,0000
0.0000
0.00000
0.0000
Sqrt Tatal tnken Tocal
0.0000
0.0000
0.0000
0,0000
Log loken/ / Log sent
0.0000
0.0000
0.0000
0,0000
Avt token sent
0.0000
0.0000
0,0000
0,0000
Avr Syll scnt
0.0000
0.0000
0.0000
0.8000
Ave Syll token
0.0000
0.0000
0.0000
0.0000
ing POS complexities of human and neural model text continuations. Overall, the finetuned GPT-2 model for general story generation obtained the least number of complexity features that are sig- nificantly different from the prompt with 31 (16 features non-significant). This is followed by the human -eeerateee continuations with 33 (14 fea- tures non-significant), the baseline GPT-2 with 37 (10 features non-significant), and GPT-2 model trained from uniformly complex data coming in last with 41 (6 features non non-significant). One clear observation that can be derived from the POS fea- tures is that all non-significance from the finetuned GPT-2 model for uniformly complex data over- laps with the continuations from humans and the finetuned GPT-2 model for general story genera- tion, especially for features related to the average use of adjective words, content words, and coordi-
Asr Chars sent
0.0000
0.0000
0.0000
Avr Chars token
0.0020
0.0000
0.0000
0,1281
Table 1: Shallow based features.
4.2 Traditional Formula-Base Features
formula-based features for readability assessment also stem from a combination of surface-based fea- tures such as word length, sentence length, and occurrence from a dre-eef dictionary of words. We covered 6 metrics such as Flesch-Kincaid (Kin- caid et al., 1975), New Automated Readability In- dex (NARI) (Smith and Senter, 1967), Coleman- Liau (Coleman and Liau, 1975), SMOG (Mc Laugh- lin, 1969), Gunning-Fog (Gunning et al., 1952), and Linsear (Klare, 1974). Due to their case of use, formulas such as the Flesch- Kincaid Reading

---

nating conjunctions. We infer that the additional fine-tuning process done for neural models encour- ages some level of uniformity of usage of the said grammatical entities with respect to prompts. Inter- estingly, fine-tuning on a general story completion dataset encourages more sensitivity to these fea- tures compared to using uniformly complex text. This is surprising one might expect increased consistency in the generated output when trained on such data. This result further highlights the need for the evaluation of complexity in language model outputs and for this task, which is non-trivial to solve.
tails 5 type-token ratio (TTR) based measures: sim- ple type-token ratio (O'Loughlin, 1995). , corrected type-token ratio (Carroll, 1964), bilogarithmic type- token ratio (Herdan, 1960), Uber index (Dugast, 1978), and simple lexical diversity. Type token features provide a quantified measure of unique word types (ex., combination of nouns, verbs, ad- jectives, and adverbs) normalized by the total num- ber of words in segment of a language. The variations of TTR have been studied over the years to minimize the effects of sentence length when calculating the values (Herdan, 1960; Tweedic and Baayen, 1998). From the results, the Uber index and the corrected TTR metric are the only two non-significant variables from texts produced by humans and uniformly finetuned GPT-2, respec- tively. This may suggest fine-tuning the GPT-2 model with uniformly complex data may observe some level of uniform lexical diversity with respect 0.0118
Feature
Human GPT2
GPT2-DF
GPT2-UF
Avr Noun POS sent
0,0000
0.0194
0.0000
0.0000
Ave Noun POS loknn
0.0000
0.0000
0.0000
0.0000
Noun POS to Adj POS
0.0002
0.1467
0.0025
0.0000
POS Verb POS
0,0000
0.0000
0,0000
0,0000
Noun POS Advrb POS
0.0000
0.0000
0.3215
0.0000
Noun POS SubriConj
0.0000
0.0000
0,0096
0.0000
Nown POS CordConj
0.0000
0.0814
0.00022
Ave Verb POS sent
0,1531
0.0000
0.0000
0.0000
to the prompt, but not for all possible metrics, and
Ave Vorb POS token
0.0000
0.6279
0.0000
0.0000
Verb POS to Adj POS
0,0244
0.0006
0.0000
0.0006
further highlights the need for the exploration of
Vort POS Noun POS
0.0000
0.0000
0.0000
0.0000
methods of generating uniform complexity in text
Verb POS to Advrb POS
0,2302
0.0000
0.0000
0.0123
Verb POS to SubrdConi
0,0000
0.0000
0,00000
0.0000
generation.
Verb to CardConj
0.0000
0.0000
0.00000
0.0000
Ave Adj POS sent
0.0557
0.0735
0.0000
0.0030
token
0,7609
0.0013
0.6105
0.0249
Feature
Human
GPT2
GPT2-DF
GPT2-UF
Adj POS to Noun POS
0,1982
0.0027
0.0000
0.0030
Adj POS to Verh POS
0.0000
0.0092
0.0024
0.0000
Simple TTR
0.0000
0.0000
0.9000
0.0000
Adj POS to Advrb POS
0.0000
0.0000
0.0506
0.0000
Correlated TTR
0.0090
0,0109
0,0000
0.5532
Adj POS to SuhrdConj
0,0000
0.0030
0,0408
0.0000
Bil oggarihhiiic TTR
0.0000
0.0000
0.8000
0,0000
Adj POS to CordConj
0.0000
0.0011
0,5120
0.6421
Uber Index
0.6039
0.0000
0.0000
0.0000
Ave Advrb POS sent
0.0000
0.0411
0.0000
0.0003
Lexical Diversity
0.0000
0.0000
0.0000
0,0000
Avr Advrh POS token
0.0000
0.0668
0.3566
0.0000
Advrb POS to Adj POS
0.0000
0.0726
0.0429
0.0030
Advrb POS to Noun POS
0.0000
0.0008
0,0000
0.0000
Table 4: Type token based features.
Advib POS to Verb POS Advrb POS to SuhrdCCanj Advib POS to CondCobj Ave SubrdCoon sont Avr SubriCooo ooj token
0.0000
0.0047
0.0000
0.00U0
0.0000
0.0000
0,0138
0.0000
0.0000
0.0247
0.5225
0.0000
0,2546
0.0000
0.00000
0.0030
0,4488
0.0000
0.0023
0.0000
Feature
Human
GFT2
GPT2-DF
GPT2-UF
SubrdConj POS to Adj POS
0.1985 0.0000
0.2602
0.0000
SubrdConj POS to Noun POS
SubrdCooj POS to Verb POS
0,2955 0.0000
0.00022
0.0000
0,0024
0.0000
0.0000
0.0030
Sqrd variation
Simpl Noun vuriation
0.0000
0.0004
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
SubrdConj POS to Advib POS
0.0002
0.0000
0.8923
0.0000
Corr Noun varistion
0.0000
0.0000
0.0000
0.0000
SubrdConj POS to Cordoon POS
Avr CardConj POS sent
0.0000
0.0000
0.0030
0.0000
0.2257
0.0000
0.0000
0.0000
Sqrd Verb variation
Simpl Verh variation
0.0000
0.0000
0.0000
0,0000
0.0000
0.0000
0.0300
0.6341
Avr CordConj POS token
0,2407
0.0000
0,4773
0.7789
Corr Verb variation
0,0000
0.1771
0.0000
0.0053
CordConj onj POS to POS
CordConj POS to Noun POS
CordConj POS to Adj POS
0,2194
0.0208
0.0000
0.0006
0.0030
0.0000
0.1113
0.00337
0.0000
0.0056
0.2777
0.0000
Corr Ad) variation
Simp Adj variation
Syrd Adj variation
0.0000
0.0000
0,0000
0,0000
0.0000)
0.0000
0.0000
0.0000
0.0000
0,0000
0.0000
0.0000
CordCoo) PUS SubrdConj POS
CurdConj POS to Advrb POS
0.0000
0.0004
0,0000
0.0000
0.0457
0.0220
0,0000
0.0030
Sqrd Adv variation
Simp Adv variation
0.0000
0.1610
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
Ave Content Words sent
0.0017
0.6533
0.0000
0.0000
Corr Adv variation
0.0000
0,0000
0.0000
0.0000
Ave Conteni Words loken
0.0000
0.0000
0,4892
0.1905
Avr Function Words token
0.0198
0.0000
0.0000
0.0000
Table 5: Lexical variation based features.
Avr Function Words token
0.0000
0.0000
0.0000
0.0000
Content to Function Words
0.0000
0.0000
0.0000
0.3664
Table 3: Part of speech based features.
4.5 Lexical Variation Features
In complement to calculating ratios and averages of word-level POS complexities, lexical variation can also signal difficulty via densities of unique grammatical components (Lu, 2012). Table 5 de- scribes 12 lexical variation-based features focus- ing on simple, squared, and corrected versions of unique counts of POS such as nouns, verbs, ad-
4.4 Type Token Features
Aside from looking at the average or ratio-based densities of POS tags locally, syntactic complexity can also be measured via densities of collective (more than one) POS tags per sentence. Table 4 de-

---

jectives and adverbs normalized its total in a sen- tence. From the results, only the unique adverb and verb variations from the human-generated, base- line GPT-2 model, and uniformly complex GPT-2 model obtained non-significance. Complementing the results previously seen in POS features, we further see the evidence of how fine-tuning may impact text continuations, especially on the usage of verbs that usually denote events and actions in a story.
such as average regular tree height and average feature tree height at the sentence level, while none for the GPT-2 models. This suggests that neural model-basee generated texts do not conform to the one property of syntax, such as parse tree height, when generating texts, despite the models them- selves having access to a significant amount of syntactic information.
Feature
Human
GPT2
GPT2-DF
GPT2-UF
Avr Noun phrs sent
0,0280
0,0000
0.0000
0.0000
4.6 Phrasal Features
Avr Noun phrs token
0.0000
0.0121
0.0000
Noun plus to Verb plns
0,0000
0.0300
0.0000
0.0000
Moving on to longer sequences of part-of-speech complexities, we also measure phrase-level lin- guistic features. Table 7 shows 42 phrase-based based features centering on token and sentence ratios of grammatical components such as noun phrases, verb phrases, adverbial phrases, phrases. subordinate phrases and prepositional phrases. Overall, the human continuation obtained 12 phrasal-based features, which were uniform with respect to the prompt, 13 for the base GPT-2, and 7 for the finetuned GPT-2 model for general story generation, and 8 for the finetuned GPT-2 model with uniformly complex data. From this result, we posit that not performing fine-tuning of the GPT-2 model preserves the syntactic structure of the sen- tence at phrase level but not at finegrained word level as seen in POS features in Table 3. We also see a connection as why Transformer-based mod- els are often finetuned for paraphrasing tasks to trigger the required number of lexical swaps and syntactic diversity (Wittevenn and Andrews, 2019; Krishna et al., 2020).
Noun phrs SubClaus
0.0000
0.0757
0.0222
0,8315
Noun phrs to Prep phrs
0.0047
0.0067
0.0000
0.0064
Noun phrs Adj phrs
0.0000
0.0000
0.0000
0.0000
Noun plrs to Adv plus
0.0000
0,0000
0.0000
0,0000
Avr Verb phrs sent
0,2050
0.0300
0,00018
0,0000
Verb phrs
0.0000
0.0000
0.0000
Verb plus to Noun plus
0.0335
0.0000
0.0000
0.0001
Verh phrs to SubClaus
0.0000
0.0300
0.0000
0.0005
Verb phrs to Prep phrs
0,5188
0.0168
0.0000
0,0000
Verb phrs to Adj plus
0.0000
0.0000
0.0000
0.0000
Verb phrs to Adv phrs
0.0000
0.0000
0.0000
0.9000
Ave SubCCaus sent Ave SubClaus loken
0.7199
0.0000
0.0021
0.2506
0,0003
0.0000
0.0000
SahClmu to Noun phrs
0,2813
0.0000
0.00000
0.0000
SuhCClau to Verb plus
0.0033
0.8824
0.0000
0.2659
SubClaus to Prop phrs
0.0169
0.2844
0.0000
0,0097
SubClaus to Adj plus
0,0000
0.0000
0.0000
0.0000
SuhClans to Adv phrs.
0.0000
0.0000
0,0000
0,0000
Ave Prep plus sent
0.1307
0.0300
0.0000
0,0000
Prep plas lokcn
0,2976
0.6734
0,0000
Prep phrs to Noun phrs
0.1831
0.5031
0.0000
0,0000
Prep pirs to Verb phus
0,0000
0.0000
0.0000
0.0000
Prep phrs to SubClaus
0.0000
0.0288
0.0014
0.0197
Prep phrs to Adj phrs
0.0000
0.0000
0,1481
0.0000
Prep phrs to Adv phrs
0.0000
0.0000
0.1013
0.0727
Avr Adj plus sent
0,0000
0.0029
0.2439
0,7460
Adj plas
0,0000
0.0000
0.0000
Adj phrs to Noun phrs
0.0002
0.0075
0.0000
0.0006
Adj plus Verb plns
0,3468
0.2768
0,4599
0.5966
Adj phrs SubClaus
0.0000
0.0014
0,0000
0.0026
Adj phrs to Prep phrs
0.6291
0.7919
0.0000
0.0173
Adj phrs Adv phrs
0.0000
0.0000
0.0000
Ave Adv plrs sent
0,0000
0.0003
0.0000
0.0000
Adv plass token
0,0000
0.3127
0.00000
0.0000
Adv phrs to Noun phrs
0.0000
0.9194
0.4361
0,0000
Adv phrs to Verb phrs
0.0000
0.0300
0,0000
0.0004
phrs to SubClaus
0.0000
0.7101
0.2759
Adv phrs to Prep phrs
0.0000
0.0603
0.0000
0,0000
phrs to Adj phrs
0.0000
0.0000
0.0000
Feature
Human
GPT2
GPT2-DF
GPT2-UP
Ave Tree height sent
0.0326
0.0000
0.0002
0.0000
Table 7: Phrasal based features.
Tree height loken
0.0000
0.0000
0.0000
0.0000
Ave Firee height sent
0.1125
0.0000
0.0000
0.0000
Avr Tiree height token
0.0000
0.0000
0.0000
0.0000
4.8
Psycholinguistic and Word Familiarity Features
Table 6: Syntax tree based features.
We also look at psycholinguistic and word famil- iarity variables in reading using external wordlists such as the Age-of-Acquisition (AOA) database compiled by Kuperman et al. (2012) which con- tains over 30,000 English content words and the UU by (Brysbaert and New, 2009) which is a compilation of over 74,000-word forms with fre- quency values extracted from 8,000 general films and series. These special databases contain words that are associated with various age levels that chil- dren are expected to learn when they reach the
4.7 Syntax Tree Features
Following the results of parase-based features in Table 7, analyzing the difference in parse tree depth is a unique and interesting way of measuring read- ability as done in Schwarm and Ostendorf (2005). Table 6 describes 4 syntax tree height-based fea- tures including average heights of regular and flat- tened trees per token and sentence. From the result, only the human- genurrted texts as obtained uni- formity with the prompts as evidenced by features

---

stage. The works of Vajjala and Meurers (2016) and Chen and Meurers (2016) both have leveraged on these predictors in readability assessment and text familiarity. Using the LingFeat tool, we extract over 26 token, lemma, and sentence- baated normal- izations of AOA and SubtlexUS variations. Results from Table 8 show that not single model has coin- cided with each other in terms of uniform features. The base GPT-2 model's continuations are all sig- nificantly different from the prompts with respect to the psycholinguistic features used, the finetuned GPT-2 model only obtained non non-significance from the averages of token-based features from the Subl- texUS dataset, and no uniformity is seen for the GPT-2 model finetuned with uniformly complex text. This suggests that fine-tuning for general story generation may introduce some control on the use of English content words but cannot be generalized all types of fine-tuning.
with intellectual disabilities. Table 9 shows the 10 discourse-level features extracted from the prompt-continuation pairs where the majority of the features have non-uniformity for both humans and GPT-2 models. Without any observable pattern of uniformity, this finding generally suggests that the generated continuations have dissimilar levels of dependencies with respect to the prompts and vice versa.
Feature
Homan
GPT2
GPT2-DF
GPT2-UF
Avr Entily sent
0.0000
0,0881
0.0000
0,0000
Avr Entily token
0.0000
0.0000
0.0000
0.0000
Avr Unig Fatity sent
0.0000
0.0000
0.0300
0.0000
Avr Unig Emily token
0.0000
0,0000
0,0000
0.0000
Local Coberence PA
0.0000
0.0000
0.0087
0.0000
Local Coherence PW
0.0000
0,0000
0.0087
0.0000
Local Coberence
0.0000
Local Cob Dist PA 0.0000
0.0000
0.0107
0,0000
Local
0.0000
0.0107
0,0000
Local Cob Dist PU
0.0000
0.0000
0.0300
0.0000
Table 9: Discourse based features.
Feature
Human
GPT2
GPT2-DF
GPT2-UF
ADA word sent
0,0300
0.0000
0.0000
0.0000
AOA word token
0.0000
0.0000
0,0000
0,0000
ADA Jemma sent
0.0000
0.0000
0.0000
0.0000
AOA lomma token
0,0500
0.0000
0.0000
0,0000
5 Similarities Between Human and Model Generations
AOA lomma Bird sont
0.0000
0.0000
0.0000
0.0000
AOA lemma Bind token
0.0094
0.0000
0.0000
0.0000
Bristel sent
0.0000
0.0042
0.0000
0.0000
AOA Bristol token
0.0300
0.0000
0.0000
0.0000
ADA CortKhanna
0,0000
0.0042
0.0000
0,0000
Aside from prompt-wise comparison, we also look at which linguistic complexity features from the GPT-2 models have some form of correlation with human continuations. This experiment aims to investigate which neural story generation models are potentially closer in producing more human- like text with respect to linguistic characteristics. For this, we use Pearson correlation to do the continuation-wise analysis and extract the top cor- related features described in Table 10. From the results, all GPT-2 models obtained a minimum of six features from the psycholinguistics cate- gory referencing Age-of-Acquiisition and SubtlexUS databases as top correlated features with human- generated texts. With this, we infer that even if there may be inconsistencies with respect to the prompts, the continuations from neural models for story generation have some level of similarity with human continuation against complexity features such as content word usage and word familiarity from the SubtlexUS database. This evidence is further strengthened by triple-occurring features across all GPT-2 models in the Table, which specif- ically include incidence of mas and words and tokens matched from the Sub- tlexUS databases.
AOA CortKhanns token
0.0000
0.0000
0.0000
0.0000
Su sent
0.0000
0.0000
0.0000
0.0000
SubticxlSS taken
0.0000
0.0000
0.6334
0.0000
SubtlexUS CD seat
0.0002
0.0000
0.0000
0.0000
SublixUS CD)
0,0000
0.0000
0.0000
0,0000
SubtlexUSS FREQ sent
0.0000
0.0000
0.0000
0.0000
SublieXUS FREQ loken
0,0300
0.0000
0,0182
0,0000
SubtixtUS CDL scnt
0,0000
0.0000
0.0000
0.0000
SubtlesUS CDL token
0.0000
0.0000
0.0000
0.0000
SubticUS SUBTL sent
0.0000
0.0000
0.0000
0.0000
SubileUS SUBTL loken
0.0300
0.0000
0.6334
0.0000
Subtevt.S 1.10WF
0.0000
0.0030
(10000
0.0000
SubtlexUS LgT token
0.8759
0.0000
0.0000
0.0000
SubtixUS SubLCD
0.06022
0.0000
0,0000
0.0000
SubtiexISS Suhl. token
0.0000
0.0000
0.0000
0,0000
SubtlexUS LgCD sent
0.0000
0.0000
0.0000
0.0000
SublixUUS LgCD token
0.0003
0.0000
0.0000
0.0000
Table 8: Psycholinguistics and word familiarity based
features.
4.9 Discourse Features
For the last linguistic feature set investigated, we look at discourse in the form of averages of umique and non-unique entity presence that can affect working memory load as well as local coherence distance measures which captures distribution and transitions of entities in passage (Barzilay and Lapata, 2008; Guinaudeau and Strube, 2013). Feng et al. (2009) previously applied these cognitively motivated features for assessing reading difficulty in the case of adults
lem-

---

GPT-2
GPT2-DF
GPT2-UF
SMOG
0,188
Avr Syll token
0.155
AOA word token
0.161
Avr Syll token
0.187
AOA Bristol token
0.102
AOA lemma token
0.158
SubtleXUS Lg IOWF token
0.177
AOA ContKhanna token
0.102
Adv phrs to Noun phus
0.130
SubtlexUS CD token
0.162
AQA lemma loken
0.102
Verb phrs Noun phrs
0.113
SubtlexUS SubLCD loken
0.162
AOA word token
0.097
Avr Adv POS token
0.109
SubtlexUS CDL token
0.159
SubtleXUS CD token
0.093
Avr Adv phrs token
0.108
SubtlexU'S LgCD token
0.154
SubtlexUS SubLCD token
0.093
SubtlexUS SubLCD token
0.100
Avr Verb POS token
0.120
SubtleXUS CDL token
0.090
SubtlexUS CD token
0.100
AOA lemma token
0.119
Avr Adv phrs token
0.080
SubtlexUS CDL token
0.094
AOA word token
0.116
Adv phrs Noun phrs
0.079
Avr Adj phrs token
0.093
Table 10:
Top correlated complexity features zero-shot GPT-2 and finetuned GPT-2 models with human
continuations. Features highlighted in light green denote double occurrences between two models, while features highlighted in dark green means occurrences across all models.
6 Benchmarking NLG Systems with UCTG
building complexity assessment models. Thus, we recommend user-centric approach for UCTG where the linguistic predictors that will be used for evaluation are catered towards specific groups of users and their language capabilities. We see strong promising support for this direction, as ev- idenced previous works where different word complexity predictors (e.g. syllable length) are correlated between subgroups of people with dis- tinet language backgrounds such as native vs. non- native English speakers (Gooding and Tragut, 2022; Gooding et al., 2021).
The evaluation of natural language generation (NLG) systems is an important process toward en- suring their usability, accessibility, and accuracy in practice. This research area has become one of the most active in the NLP community in recent years due to the rise of more complex and powerful gen- erative models (Gehrmann et al., 2021; : Caglayan et al., 2020). Synthesizing the results shown from Sections 4 and 5 in this study reporting the current inconsistencies of finetuned GPT-2 models in gen- erating uniformly complex texts, one may reason out the potential of UCTG as a benchmark task for existing NLG systems to gather attention and encourage work from the research community. For this proposition, we highlight three major points be- low as insights that may be considered for building more uniformly complex text generation models.
6.2 Advantages of using language-specific readability metrics.
Current disadvantages of automatic evaluation met- rics include weak correlation and reflection with human preferences well limitations consid- cring finer-grained information during assessment (ex. sentence-level processing) (Sai et al., 2021; I; Re- iter, 2018; Reiter and Belz, 2009; Stent et al., 2005), Moreover, works on expanding NLG research to- wards multilingual data have also gained signif- icant traction (Gehrmann al., 2021; Hu et al., 2020; Ponti et al., 2020). In this regard, more and more efforts from the NLG community, such as the work of Novikova et al. (2017), for instance, sup- port the proposal for data-independent automatic evaluation methods, especially for multilingual and cross cross-domain applications. While this proposition has clear benefits, there is still limited work on text complexity research the transferability of lin- guistic complexity predictors from one language to another. And so far, previous works have em- phasized the importance of still using language- specific complexity predictors such as in Imperial and Kochmar (2023) and Imperial et al. (2022) for
6.1 Advantages considering user background.
Despite being the main focus in readability assess- ment research, text complexity is century-ol. re- search area that dates back the 1920s (Thorndike, 1927), and task-agmostic multi-faceted variable that can be applied almost all language-related experiments. This makes UCTG, unlike other au- tomatic metrics restricted to certain tasks such as BLEU (Papineni et al., 2002) or METEOR (Baner- jee and Lavie, 2005) for machine translation, likely to have more impact towards accessibility and user- dependence of NLG systems if carefully consid- ered. However, the current methods for evaluat- ing text complexity are wide-ranging-as evidenced by the 160 linguistic features used in this study which can potentially be used as predictors for

---

Philippine languages, and Weiss et al. (2021) for German. Thus, for applying UCTG evaluate the uniform complexity of multilingual NLG systems, we still recommend going back to the basics, such as training models with linguistically plausible and domain-specific features as starting points.
recommendations covering the importance of user background, language dependency, and flexibility of implementation if UCTG is to be implemented as an automatic evaluation metric or benchmark task for any arbitrary generative models from here onwards.
6.3 Diversity in selecting the right text complexity metries.
Limitations
We discuss clarifying statements below regarding the scope and limitations of this study.
Finally, we emphasize that the method of bench- marking NLG systems with UCTG is not strictly prescriptive and linear as with other evaluation metrics used in text generation. The choice of se- lecting the appropriate combination of linguistic features to measure the complexity of machine- generated texts is suggestive and entirely up to the researcher's judgment. For example, one may exclusively select vocabulary-based metrics deriving from psycholinguistic features similar to the ones reported in Table for language learning tasks such as question generation as done in (Jiao et al., 2023). Moreover, a researcher can also train machine learning models from an extensive set of selected linguistic features as scorers of text com- plexity as done in Lee et al. (2021). The variety of candidate linguistic features may be selected with consideration of the domain of data used for training the generative models, its target users (as discussed previously), the language intricacies and properties the data, and the extensiveness of the evaluation of said models.
On the specific use GPT-2 models. While the possibility of using more advanced models such as GPT-4, and ChatGPT may come to one's option for nearly all generative tasks, we emphasize that the availability of these models do not disregard the rich literature of narrative generation work using GPT-2. Likewise, models such ChatGPT and GPT-4 cannot technically and freely be considered baseline models for comparison as the developers of these models provided limited transparency their documentation and a paywall for their usage.
On proving the research problem exists. We emphasize that the goal of this study is prove that the research problem, the challenge of maintaining the complexity of generated responses with respect to prompts, exists rather than proposing experimenting with a slew of controllability methods for LLMs without a proper in-depth grounding of the problem. Our empirical investigation makes use of full linguistic analysis of grompt-continuation pairs from humans and neural language models. We explore methods such as fine-tuning as the standard way of calibrating LLMs. We show that this task may be complex in nature, and fine-tuning may not be a direct solution to the problem, thus an open research opportunity for researchers in the field to explore.
7 Conclusion
this study, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark task to encourage the production of uniformly com- plex generations with respect to prompts. To fur- ther ground the existence of the problem, we per- form the most extensive investigation of linguistic properties (160 features) analyzed from genera- tions of GPT-2 models for the narrative generation task. We find clear inconsistencies with the models, whether finetuned or not, in terms of preserving the text complexity of generations with respect to the prompts used. However, we do note some sim- ilaritics in specific linguistic variables related to word familiarity and word use between humans and GPT-2 models in how they produce continua- tions to a prompt. Recognizing the potential and benefits of UCTG to increase the usability and ac- cessibility of NLG systems, we discuss three major
On experiments exclusively with English data. All experiments, findings, and insights in this work only apply to English, as evidenced by the language of the datasets used. Thus, our findings may not generalize similar research derived from this work is to be done with other languages using other models, such as those trained with multilingual data.
On varying complexities with increasing length. Our experiments, particularly with the generation

---

setup as described in Section 3, focus on a specific limit of tokens (minimum 150 and maximum 300) for uniformity of analysis across models. The complexity of a text may not be directly linear when substantially longer passages are generated beyond our experimentation. We leave the fine-grained exploration of measuring complexities of text segments and chunks to future work.
01-MLA-CCIT), and the Study Grant Program of National University Philippines.
References
Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE) pages 1549- 1564, Hong Kong, China, Association for Computa- tional I Linguistics.
On use of variables beyond linguistic features. Our experiments and findings specifically focus on possible inconsistencies found in texts generated by LLMs using linguistically motivated features such as vocabulary use, syntax, discourse, and part of speech. We note that there are other forms of analysis done previously using information theory concepts such as entropy and uniform information density principle on tasks such as dialogue-basee data (Giulianelli and Fernández, 2021; Giulianelli et al., 2021). We currently have no proof how these concepts relate or connect to inconsistencies associated with linguistic features but leave this to future work.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit lyyer. 2020, STO- RIUM: A Dataset and Evaluation Platform for Machine-in-the-Lowwe Story Generation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6470-6484, Online. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACI. Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sammarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Ethics Statement
The datasets used, WRITINGPROMPTS NEWSELA, and ELG, are all available for researchers through the links provided, with some minor information needed to provide the purpose of use. In addition, we do not involve any human participants in this study as it is centered on text complexity which is an automatic evaluation by nature. Similar to the GPT-2 model finetuned by DeLucia et al. (2021), the GPT-2 model finetuned with uniformly written texts (NEWSELA, ELG) has been made available in Huggingface.
Regina Barzilay and Mirella Lapata. 2008, Modeling local coherence: An entity- approach. Compu- tational Linguistics, 34(1): 1-34.
Scott F Beers and William E Nagy. 2009. Syntactic complexity a predictor of adolescent writing qual- ity: Which measures? Which genre? Reading and Writing, 22(2):185-200
Mark Breuker. 2022. CEFR Labelling and Assessment Services. In Earopean Language Grid: A Language Technology Platform for Multilingual Europe, pages 277-282. Springer International Publishing.
Acknowledgements
Mare Brysbaert and Boris New. 2009. Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4):977~ 990.
We thank the anonymous reviewers for their con- structive feedback and the ACs, SACs, and PCs for their appreciation of this work. We also thank Alexandra Delucia and Ekaterina Kochmar for their valuable feedback on the initial version of this work and Mark Townsend for the assistance with configuring the experiments with the Hex GPU cloud of the Department of Computer Science at the University of Bath. JMI is supported by the UKRI Centre for Doctoral Training in Account- able, Responsible and Transparent AT (ART-AT) [EP/S023437/1] of the University of Bath, the NU Research Faculty Program (Project ID: F-2
Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious of language generation evalua- tion metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2322-2328, Barcelona. Spain (On- line). International Committee on Computational Lin- guistics.
John B Carroll. 1964. Language and Thought. Founda- tions of Modern Psychology Series.

---

Haw-Shiuan Chang, Jiaming Yuan, Mohit lyyer, and Andrew McCallum. 2021. Changing the mind of transformers for topically-controllbbe language gen- cration. In Proceedings of the 16th Conference of the European Chapter of the Association for Compula- tional Linguistics: Main Volume, pages 2601-261 Online. Association for Computational Linguistics.
Durmus, Ondfej Dušck, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jemnie, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan- Major. Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo. Salomey Osei. Ankur Parikh, Laura Perez-Beltrachin Beltrachini Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation. its evaluation and metries. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
Xiaobin Chen and Detmar Meurers. 2016. Characteriz- ing text difficulty with word frequencies. In Proceed- ings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 84-94, San Diego, CA. Association for Computational Lin- guistics.
Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283.
Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoc. 2021. Decoding methods for neural narrative generation. In Proceedings of the 1st Work- shop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 166-185, Online. Association for Computational Linguisties.
Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th An- mual Meeting of the Association for Computational Linguistics, pages 199-2066 Philadelphia. Pennsylva- nia, USA. Association for Computational Linguistics.
William H DuBay, 2004. The Principles of Readability. Online Submission.
D Dugast, 1978. Sur quoi se fonde la notion d' étenduee théoretique du vocabulaire? Le Francais Moderne, 46:25-32.
Edward E Gickling and David L Armstrong. 1978. Lev- els of Instructional Difficulty as Related to On-task Behavior, Task Completion, and Comprehension. Journal of Learning Disabilities, 11(9):559-55) 566.
Angela Fan, Mike Lewis. and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Mario Giulianelli and Raquel Fernández. 2021. Analysing human strategies of information transmis- sion as a function of discourse context. In Proceed- ings of the 25th Conference on Computational Nat- ural Language Learning, pages 647-660, Online. Association for Computational Linguistics.
Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readabil- ity assessment. In Proceedings of the 12th Confer- ence of the European Chapter of the ACL (EACL 2009), pages 229-237, Athens, Greece. Association for Computational Linguistics.
Mario Giulianelli, Arabella Sinclair, and Raquel Fer- nández. 2021. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pages 8271-8283, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221.
Irene C Fountas and Gay Su Pinnell. 1999, Matching Books to Readers: Using Leveled Books in Graded Reading, K-3. ERIC.
Sian Gooding, Ekaterina Kochmar, Seid Muhie Yimam, and Chris Biemann. 2021. Word complexity is in the eye of the beholder. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4439-4449 Online. Association for Computational Linguisties.
Thomas François and Eleni Miltsakaki. 2012. Do NLP and machine learning improve traditional readability formulas? In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations. pages 49-57, Montréal, Canada. Association for Computational Linguistics.
Sian Gooding and Manuel Tragut. 2022. One size does not fit all: The case for personalised word complexity models. In Findings of the Association for Compu- tational Linguistics: NAACL 2022, pages 353-365, Seattle, United States. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustuh Dhole, Wanyu Du, Esin

---

Jian Guan and Minlie Huang. 2020. UNION: An Un- referenced Metric for Evaluating Open-ender Story Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing J.MNLP),Re pages 9157-9166, Online. Association for Computational L Linguistics.
Seattle, Washington. Association for Computational Linguistics.
Ying Jiao, Kumar Shridhar, Peng Cui, Wangchunshu Zhou, and Mrinmaya Sachan. 2023, Automatic ed- ucational question generation with difficulty level controls. International Conference on Artificial Intelligence in Education, pages 476-488
Rowena C. Guevarra. 2011. Development of Filipinoo
text readability index.
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation Of New Readability Formulas (Automated Readability Index, Fog Count And Flesch Reading Easc For- mula) For Navy Enlisted Personnel. Technical report, Naval Technical Training Command Millington TN Research Branch.
Camille Guinaudeau and Michael Strube. 2013. Graph- based local coherence modeling. In Proceedings of the 51st Anmual Meeting of the Association for Computctional Linguistics (Volume 1: Long Papers), pages 93-103, Sofia, Bulgaria. Association for Com- putational Linguistics.
Robert Gunning et al. 1952. Technique of Clear Writing. McGraw-Hill Hill.
George R Klare. 1974. Assessing Readability. Reading Research Quarterly, pages 62-102.
Michael Heilman, Kevyn
Jamie Callan, and Maxine Eskenazi. 2007. Combining lexi- cal and grammatical features to improve readability measures for first and second language texts. In Hu- mun Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 460-467, Rochester, New York. Association for Computational Linguistics.
Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as para- phrase generation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online. Asso- ciation for Computational Linguistics.
Victor Kuperman, Hans StadthagenGGonzalez and Marc Brysbaert. 2012. for 30,000 English words. Behavior research meth- ods, 44(4)9978-990.
ratings
Gustav Herdan. 1960. Type-token Mathematics: A Text- book of Mathematical Linguistics. Mouton.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig. Orhan Firat, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi- task Benchmark for Evaluating Cross-lingual Gener- alization. International Conference on Machine Learning, pages 4411 14214 PMLR.
Bruce W. Lee, Yoo Sung Jang, and Jason Lee, 2021. Pushing on text readability assessment: A trans- former meets handcrafted linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages I 10669- 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Chich-Yang Huang, Saniya Naphade, Kavya Laalasa Karanam, and Tio Kenneth Huang. 2023. Con- veying the predicted future to users: A case study of story plot prediction. In The AAAI-23 Workshop on Creative AI Across Modalities.
Xiaofei Lu. 2012. The Relationship of Lexical Richness to the Quality of ESL Learners' Oral Narratives. The Modern Language Journal, 96(2):190-25) 208.
Joseph Marvin Imperial and Ekaterina Kochmar. 2023. Automatic Readability Assessment for Closely Re- lated Languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5371 5386, Toronto, Canada. Association for Computa- tional Linguistics.
G Harry Me Laughlin. 1969. SMOG Grading A New Readability Formula. Journal of reading, 12(8):639- 646.
Danielle McNamara, Scott A Crossley, and Philip M McCarthy. 2010. Linguistic Features of Writing Quality. Written communication, 27(1):57-86.
Joseph Marvin Imperial and Ethel Ong- 2021. Diverse Linguistic Features for Assessing Reading Difficulty of Educational Filipino Texts. In 29th International Conference on Computers in Education Conference, ICCE 2021, pages 51-56. Asia-Pacific Society for Computers in Education.
Jekaterina Novikova, Ondřej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why we need new evaluation metries for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2241-2252, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Joseph Marvin Imperial, Lloyd Lois Antonie Reyes. Michael Antonio Ibanez, Ranz Sapinit, and Mo- hammed Hussien. 2022. A baseline readability model for Cebuano. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 27-32,
Kieran O' O'Loughlin. 1995. Lexical density in candidate output on direct and semi-direct versions an oral proficiency test. Language Testing, 12(2):217-237.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 843-861, Hong Kong, China. Asso- ciation for Computational Linguistics.
Edgar A Smith and RJ Senter. 1967. Automated read- ability index, volume 66. Aerospace Medical Re- search Laboratories, Acrospace Medical Division.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-20 2376, Online. As- sociation for Computational Linguistics.
Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating Evaluation Methods for Generation in the Presence of Variation. In CICLing, volume 2005, pages 341-351. Springer.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella- Micke, and Mohit Tyyer. 2021. Do long-range lan- guage models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807- 822, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learners. Ope- nAI Blog. 1(8):9.
Ehud Reiter. 2018. A structured review of the validity of BLEU. Computationnl Linguistics, 44(3):393 4011.
Edward Lee Thorndike. 1927. The teacher's word book. Teachers College, Columbia University.
Ehud Reiter and Anja Belz. 2009. An investigation into
the validity of some metrics for automatically evalu- ating natural language generation systems. Computa- tional Linguisties, 35(4):529-555
Fiona J Tweedie and R Harald Baayen. 1998. How Variable May a Constant Be? Measures of Lexical Richness Perspective. Computers and the Human- ities, 32(5):323-352.
Melissa Roemmele. 2021, Inspiration through Ob- servation: Demonstrating the Influence of Gener- ated Text on Creative Writing. arXiv preprint arXiv:2107.04000
Sowmya Vajjala and Detmar Meurers. 2016. Readability-Basee Sentence Ranking for Eval- uating Text Simplification. arXiv preprint
Melissa Roemmele, Andrew S Gordon, and Reid Swan- son. 2017. Evaluating Story Generation Systems Using Automated Linguistic Analyses. In SIGKDD 2017 Workshop on Machine Learning Creativity. pages 13-17.
16X::1600300609
Zarah Weiss, Xiaobin Chen, and Detmar Meurers. 2021. Using Broad Linguistic Complexity Modeling for Cross-Lingual Readability Assessment. In Proceed- ings of the 10th Workshop on NLP for Computer Assisted Language Learning, pages 38-54, Online. LiU Electronic Press.
Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. Perturbation CheckLists for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7219-7234, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Bernard Welch. 1947. The Generalization of STU- DENT'S Problem When Several Different Population Variances Are Involved. Biometrika, 34(1 (1-2):28-333 35.
Sam Witteveen and Martin Andrews. 2019. Paraphras- ing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Trans- lation, pages 215-220, Hong Kong. Association for Computational I Linguisties.
Carolina Scarton, Gustavo Paetzold, and Lucia Spe- cia. 2018 Text Simplification from Professionally Produced Corpora. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung. Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON--NTTLL Control- lable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2845 Online. Association for Computational Linguistics.
Sarah Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting of the Association for Compu- lational Linguistics (ACL '05), pages 523-530, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Wei Xu, Chris Dallison-Burch, and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 3:283-297.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better

---

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Buree 2016. Optimizing sta- tistical machine translation for text simplification. Transactions the Association for Computational Linguistics, 4:401-415.
Zhexin Zhang, Jiaxin Wen, Jian Guan, and Minlie Huang. 2022. Persona-guidke planning for control- ling the protagonist's persona story generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3346-3361, Seattle,, United States. Association for Computational Linguistics.
A Uniform Barplot Visualization per Linguistic Feature

---

AOA Bristol sent
AOA Bristol token
AOA CortKhanna sent
CortKhanna token
1.0
1.0
1.0
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
9pt2
gPt2 of
prompt
human
upt2 di
BP22 uf
prampt
human
GPC
9pt2 di
got2 cd
prompt
human
gpc2
9pc
0pt
ADA lemma Bird sent
AQA lemma Bird token
lemma sent
AOA lemma token
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.2
0.2
0.2
0.2
H
0.0
0.0
0.0
prompt
human
0142
aunn uf
prompt
human
9l42
gint2 dr
brompt
human
0B12
gRt2 di
gA12 of
promgt
human
unf2
gme2. df
0M12
0142
got2
AOA word sent
AOA word token
Adj POS to Advrb POS
Adj POS to CordConj
1.0
1.0
1.0
1.0
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
9pt2
prompt
human
gat2
ppt2 df
ant2 of
prompt
human
8n2
prompt
human
gntz
ant2. of
BP
gpt2
gnt
Adj POS to Noun POS
Adj POS SubrdConj
Adj POS to Verb POS
Adj phrs to Adv phrs
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
Iumma
9pt2
gpdz uf
prompt
human
9pk2 gnt2.0 df
amt2. tff
prompt
human
9012
gpt2.. df
go12. of
prampt
human
9nt2 gmtz.. df
got2.o
gpt2.
phrs to Noun phrs
Adj phrs to Prep phrs
Adj phrs to SubClaus
Adj phrs to Verb phrs
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
prompt
hamman
99a2
prompt
tuman
9012
prampt
human
get2
9pt2. di
promet
human
gpt2
got2.
gM2
9nF2
g012,
gn22.
gnt2.
gpt2,

---

Adv phrs to Adj phrs
Adv phrs to Noun phrs
Adv phrs to Prep phrs
Adv phrs to SubClaus
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
Mpt2
gAt2 df
prompt
human
gat2
upt2 di
0Pt2 ud
prampt
human
9pt2 di
prompt
human
9nt2
Bpt2 of
Bpc
go12
gpt2
Adv phrs to Verb phrs
Advrb POS to Adj POS
Advrb POS to CordCobj
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
human
0142
prompt
humani
UA22 dr
BRt2 uf
brompt
human
9pt2
got2 of
gla2 uf
prompt
Fuman
BPF2
am22 di
dni2 uf
0A12
gpt2
Avr Adj POS sent
Avr Adj POS token
Advrb POS to SubrdConj
1.0
1.0
1.0
1.0
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
GB12
goc2 of
prompt
human
upt2
BR22 df
prompt
human
8n2
mol di
prompt
human
gnt2
0nt2 df
9oi2
mni2
1o12
gnt2
Aur Ad] phrs sent
Aur Adj phrs taken
Avr Adv phrs sent
Avr Adv phrs token
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
Inuman
9pt2
aadz uf
prompt
human
9pk2
gnt2.0 df
gmt2. tff
prompt
human
9012
gpt2. df
go12 uf
prampt
human
9n12 gmtz.. df
got2.
gpt2
Aur Advrb POS sent
Aur Advrb POS token
Avr Chars sent
Avr Chars taken
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
prompt
harman
gta2
prompt
tuman
9e12
prampt
human
get2
9pt2. di
promet
human
gpt2
gwt dr
got2.
gM22,0
9n22
9012,
gn12.
gpt2,

---

Avr Content Words sent
Words token
Avr CordCon) POS sent
Avr CordConj POS token
1.0
1.0
1.0
0.8
0.8
0.3
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
gnr2
9pt2 or
gpt2. ud
prompt
human
Gp12
GR2 of
prompt
hurman
got2
9nt2 di
un82 ud
prompt
human
Gmf2
gpt2. ud
giz2
8Rt2
Entity
Avr Entity taken
height sent
height
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
humaan
9ra2
99t2 uf
prompt
human
9pt2
9aa2 di
gna2 uf
prompt
human
gat2 uf
prompt
human
got2 gma2. df
9ma2
99t2.
9oa2
Avr Noun POS sent
Aur Noun POS taken
Avr Noun phrs sent
Aur Noun phrs token
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
prompt
himan
0122
gpt2. dr
gpt2 uf
prompt
human
0jt2
upt2 di
gpt2 tf
prompt
human
GRL2
gpt2 di
get2. u
prompt
human
gatz
gpt2 dr
gpt2
Avr Prep phrs sent
Avr Prep phrs token
Avr SubClaus sent
Avr SubClaus token
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
0162
prompt
human
Upt2
upt2 df
gat2 of
prompt
human
0122
9p12 di
get2 of
prompt
human
gra2 ger22 df
9m12.0m
9pt2.1.
9pt2.
Avr SubrdConj sent
Avr SubrdConj token
Avr Syll sent
Avr Syll token
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.0
0.0
0.0
prompt
humann
Upt2
1Rt2. or
prompt
human
9p12
1882. df
prompt
human
up12
apt2 dr
promet
human
gpt2
gpt2.. or
gRt2. ur
1ua2
9pt2,
Spt2

---

Avr Tree height sent
Avr Tree height token
Avr Uniq Entity sent
Avr Uniq Entity token
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.2
prompt
human
0p22
9pt2 of
gpe uf
prompt
human
0pk2 9pt df
9pt2 uf
mrampt
human
0m12 goe2 di
Bpt2 of
prompt
human
0pt2 opt2. df
gpr2
Verb POS
Avr Verb POS token
Avr Verb phrs
Avr Verb phrs token
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
97A2
prompt
human
glt2
get2. uf
prompt
human
Gmt2
prompt
human
gnt2
uRe2. of
gpt2
gpt2.
BP12
gnt2.
Avr taken sent
BiLogarithmic TTR
Coleman-Liau
1.0
1.0
1.0
Content Function Words
1.0
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
human
gpt
Up2 uf
prompt
human
gpt2 9pt2.0 dr
an uf
prompt
human
gb12
ppt di
1o12 u
prompt
human
am22 of
0p
9pt2.
CordConj POS
CordConj POS Advib POS
CordConj POS POS
1.0
CordCan POS to SubrdCanj POS
1.0
0.8
0.8
0.5
0.5
0.2
0.2
0.0
0.0
0.0
prompt
huran
got2
gpt2 df
9pt2
prompt
humaan
g012
2012
gat2.
prempt
human
9pr2
spe2.
gpt2 uf
prompt
human
uma gec2 of ge22
Adj variation
Corr Adv variation
Noun varlation
CordConj
0.8
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0,0
0.0
prompt
hunan
9nt2
9po2. of
gp22.. uf
promit
human
gp12
prompt
human
unt2
upt2 di
prompt
human
gpt2
gnt2 df
imt2.u.
got2,
1812.
got2.

---

Corr Verb variation
Correlated TTR
Flesch Flesch-Kincaid
Gunning-FFii Fog
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
0pt2
gAt2. df
prompt
human
gjtt2
upt2 di
BPt2 uf
prampt
human
9pt2 di
go12 cd
prompt
human
0nt2
gpt2. of
Bpt2
gpt2
Lexical Diversity
Linsear
Coh Dist PA
Local Dist
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
0142
gaa2 uf
pronpt
human
0pt2
UP22 df
BRt2 uf
brompt
human
um12
gRt2 di
gA12 of
prompt
human
gna2
gmt2. df
0A12
gat2
Local Coh Dist PW
Local Coherence PA
Local Coherence PU
Local Coherence PW
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
99t2
aP or
prompt
human
gatz
upt2 df
8812 of
prompt
human
gn2
got di
prompt
human
ontz
anz2 of
gpt2
gnt2
Log taken Log sent
NARI
Noun POS to Adj POS
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
Iuuman
9pt2
gpt2. uf
prompt
human
9pk2
gmt2.0 df
amt2. tif
prompt
human
9012
gpt2.0 df
go12. of
prampt
human
Bpt2
gRR2 df
getz if
got2.o
Noun POS to CordConj
Noun POS to SubrdCon)
Noun POS Verb POS
Noun phrs to Adj phrs
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
prompt
hamann
pompt
human
0812
spt2 di
gat2 of
prampt
human
gat2
gntz df
promet
human
gpt2
got2.
g012
got2,
ant2.
gpt2,

---

Noun phrs to Adv phrs
phrs Prep phrs
SubClaus
phrs phrs
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
prompt
human
unt2
upt2 of
prompt
human
0nd2
go12 dr
gpt2 ur
prompt
human
0mi
mpt2 of
upt2.
8Pc
gpt2
gpt2
Prep phrs to Adj phrs
Prep phrs to Adv phrs
Prep phrs to Noun phrs
Prep phrs SubClaus
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
0142
yann uf
prompt
human
0pt2
UP22 df
brompt
human
9M12
09A2. df
BP22 uf
prompt
human
gnf2
gme2. df
gm12
URR2
g0t2
Prep phrs to Verb phrs
SMOG
Simp Adj variation
Simp Adv variation
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
BRt2
BPt2 dr
prompt
human
gpt2
upt2 df
upt2 of
prompt
human
Bpa di
prompt
human
untz
gpt2 of
8PL2
gpt2
gpt2
Simpl Noun variation
Simpl Verb variation
Simple TTR
Sqrd Adj variation
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
Iuuman
9pt2
prompt
human
9pk2
gnt2.0 df
gm12 tf
prompt
human
9012
gpt2... df
got2 c
prompt
human
gnt2
gmtz.. df
got2
gpa2.5
gpt2
Sqrd Adv variation
Sqrd Noun variation
Sqrd Verb variation
1.0
1.0
Sqrt Total token x Total sent
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
prempt
human
9942
prompt
human
9P12
9nt2.s. df
prampt
human
gRt2
9pt2 di
prompt
human
Bnt2
umz of
Bp
gpt2,
2p12
0912.
gm12.

---

SubClaus to Adj phrs
SubClaus to Adv phrs
SubCiaus Noun
SubClaus to Prep phrs
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
Bpt2
gAt2 df
prompt
human
got2
upt2 di
BPt2 uf
prompt
human
git2
gp12 dr
gp12 ur
prompt
human
Jntz
apt2. of
gpt2
gpt2
SubClaus phrs
1.0
SubrdConj POS to Adj POS
SubrdConj POS to Advrb POS
SuardiConj POS CordCanj POS
1.0
0.8
0.8
0.2
0.2
0.0
0.0
0.0
prompt
human
Bre2
0m12
gpa2 uf
prompr
human
ypt2
9pt2. or
spt2 or
prompt
hurman
unz
9mi2 ok
prompt
human
gocz of
gPi2 of
SubtlexUS CD sent
SubtlexUS CD token
SubrdConj
CunrdConj POS to Verb POS
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
prompt
human
g012
Bm22 df
amiz ad
prompt
humann
sptt am2 df
prompt
human
gn2
Git2 di
prompt
human
gntz
ant2 of
2012
gnt
SubtlexUS CDL sent
SubtlexUS CDL taken
SubtlexUS FREQ sent
SubtlexUS FREQ token
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
prompt
Iumma
9pt2
prompt
human
9pt2
gnt2.0 df
umt2. tff
prompt
human
9012
gpt2.d di
go12. of
prompt
human
go12
gptz. df
got2.
gpt2.
mpt2,
SubtlexUS Lg10WF sent
SubtlexUS Lg10WF token
SubtlexUS LgCD sent
SubtlexUS LgCD token
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
8812
9pt2 df
prongt
humann
gnt2
mpt2. or
sptb, uf
prampt
human
get2
9pt2. di
promet
himan
gpt2
gpt2 df
gn12.
gpt2.

---

SubtlexUS SUBTL sent
SubtlexUS SUBTL token
SubtlexUS SubLCD sent
SubtlexUS SubLCD
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
9pt2
prompt
human
upt2
BP42. of
prompt
human
9992
prompt
human
0Ri2
gPt2.
gpc2
Bpt
upt2.
upt2
SubtlexUS sent
SubtlexUS
Total
Total
Uber
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
0142
apn2 uf
prompt
human
0pt2
UA22 dy
BRR2 uf
prompt
human
gotz
umz ak
prompt
human
unt2 gme2 df
0A12
0pt2
gat2
Verb POS to Adj POS
Verb POS to CordConj
Verb POS to Noun POS
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
prompt
human
9pt2
BP of
prompt
human
upt2
BR22. df
prompt
human
8n2
prompt
human
gp12
gnt2 of
mnt2
gpt2
got,
Verb POS to SubrdConj
Verb phrs Ad] phrs
Verb phrs Adv phrs
Verb phrs to Noun phrs
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
prompt
human
9012
gpt2 up
promitt
human
9pk2
gnt2.0 df
gmt2. tff
prompt
human
9012
gpt.. di
got2. uf
prompt
human
9pt2
gnii... of
gnt2
gnf2.
Verb phrs phrs
Verb phrs to SubClaus
0.8
0.8
0.5
0.5
0.0
0.0
prompt
hamann
prompt
hurman
gpt2
gRt2. di
got2.
gR12
gpt2


Reviews:
[{"review_id": "Qd0kvsKQOQ", "reviewer": "Reviewer_kTEu", "summary": "", "questions": "* It’s unclear if these attributes are stable across human texts in general, not just in the prompt vs. continuation setting. Some sort of temporal analysis (i.e., over different chunks of a given text) would have been nice, both in the human and model generation settings. Like does complexity monotonically increase/decrease?\n* What are the directions of change for the various metrics? Are they increasing or decreasing?\n* It’s unclear what the values being reported in the tables actually are. Are they t-statistics, p-values,...?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s very positive feedback and the score given to our work. Please find our responses to your individual questions below. We hope that our responses are able to sufficiently address and clarify your questions and concerns. We have taken note of all suggested improvements to our paper in terms of presentation, grammar, references, and content revisions.\n\n1. *\"The authors don’t offer a single comprehensive complexity metric. The practical implications of this paper are not obvious.\"* - As discussed in part of the Introduction (second page), unlike other metrics such as semantic similarity, text complexity is a characteristic that can be measured by a variety of content-based and linguistic factors covering syntactic, discourse, vocabulary, frequency, etc. Thus, compressing all of them into one complexity metric may trivialize the task. We hope that retaining the various features that makeup text complexity will help in alleviating this problem. We also reflect this in Section 5 - Going Forward, where we recommend that while researchers have the freedom to implement any linguistic metric to test for UCTG, they would also have to consider factors such as user background and properties of the language they will work on. Our tone for this recommendation inclines more on being suggestive of the benefits rather than prescriptive. \n\n2. *\"The analysis is limited to a relatively small language model, and it's unclear if the results scale to the larger models that are more often used today for generations.\"* - We understand the valid concern of the reviewer on the limitations of specifically using GPT-2. We would like to emphasize that our work is derived from the rich literature of story generation studies using GPT-2, as acknowledged in the Limitations section. Moreover, GPT-2 is still one of the open-source OpenAI models that the public can access freely. Thus, our contribution can be beneficial for future works studying LLMs, especially if factors such as scale, interpretability, and linguistic inconsistencies are considered.\n\n3. *\"Looking at more fine-grained segments would arguably be more informative, to see if models are able to maintain complexity across their generations. // Some sort of temporal analysis would have been nice, both in the human and model generation settings.\"* - We would like our experiments to complement previous works that have also used the same dataset as WritingPrompts (ex. Fan et al, 2018; Delucia et al 2021) in which the narratives are not excessively long (maximum 100-256 words). However, the reviewer raised a very good point, and while we cannot bring new results here due to time constraints. We will definitely explore this approach for our next revisions of the paper.\n\n4. *\"What are the directions of change for the various metrics? Are they increasing or decreasing?\"* - This is a great recommendation. While we conducted a two-tailed t-test, which is inherently non-directional, we can provide some information of direction by comparing the mean of each feature for the continuation, whether they are lesser or greater than the prompt. We can do this by adding arrow lines (red for decrease and green for increase) for each column per feature in the tables after revision.\n\n5. *\"There is no discussion of related work\"* - We explicitly did not add an RRL section because we wanted to scatter the comparison of previous works across all parts of the paper with their respective discussions. Previous works on evaluating generations of LLMs closely similar to ours have been cited in Section 1 - Introduction, Section 2 - Generation Setup, and Section 5 - Going Forward. We can include a separate RRL section for our next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is very clearly written and well-motivated\n* Text complexity is an interesting aspect of generation that hasn’t received much attention. The authors convincingly argue that it should be taken into consideration\n* The authors use a comprehensive set of complexity metrics in their analysis\n", "reasons_to_reject": "* The authors don’t offer a single comprehensive complexity metric; while this might be difficult, its unrealistic to ask the NLG community to evaluate 176 different attributes of texts for every given system. Therefore, the practical implications of this paper are not obvious\n* The notion of “uniform” in this work is limited to only the first and second half of a text, which is quite a coarse-grained analysis. Rather, looking at more fine-grained segments would arguably be more informative, to see if models are able to maintain complexity across their generations\n* The analysis is limited to a rather small language model and its unclear if the results scale to the larger models that are more often used today for generation\n* There is no discussion of related work\n", "questions_to_authors": "* It’s unclear if these attributes are stable across human texts in general, not just in the prompt vs. continuation setting. Some sort of temporal analysis (i.e., over different chunks of a given text) would have been nice, both in the human and model generation settings. Like does complexity monotonically increase/decrease?\n* What are the directions of change for the various metrics? Are they increasing or decreasing?\n* It’s unclear what the values being reported in the tables actually are. Are they t-statistics, p-values,...?\n", "paper_topic_and_main_contributions": "This paper discusses the evaluation of text complexity and whether standard language models are able to maintain the linguistic complexity of the context on which they condition. The work discusses different classes of metrics that can be used to assess the complexity of text. It then performs a large analysis of models trained under different settings/objectives, and evaluates whether text complexity is significantly different in the prompt vs. continuation. They find that according to most metrics, the complexity of a prompt is often not maintained by the generation models; this is true under the different model settings tested. However, this seems to also be the case for human continuations of the same prompt.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* Entropy Rate Constancy in Text. Dmitriy Genzel and Eugene Charniak. 2002\n* Is Information Density Uniform in Task-Oriented Dialogues? Mario Giulianelli, Arabella Sinclair, Raquel Fernández. 2021\n* Analysing Human Strategies of Information Transmission as a Function of Discourse Context. Mario Giulianelli and Raquel Fernández. 2021.\n"}}, {"review_id": "ouR2WtbXW8", "reviewer": "Reviewer_bS2i", "summary": "", "questions": "(a) (28) Is this just true for story writing or for text generation in general?\n\n(b) (50) What’s the difference between text complexity and readability? Does it matter here?\n\n(c) I would mention in the intro that you actually provide tests to quantify whether uniformity is observed for humans too!\n\n(d) (171) One problem with this model is that you assume that humans do, in fact, produce uniformly complex material during generation. I think this assumption should be stated clearly. While I agree with the assumption, one problem is that the test you provide seem to indicate differently.\n\n(e) (192) A number of other decoding strategies have emerged in recent years, including locally typical sampling (Meister et al., 2022) and eta/trunctation-sampling (Hewitt et al., 2022). I wonder if the results reported here generalize across decoding strategies.\n\n(f) (Table 1) It’s never stated what the numbers in the table are. I assume that they are the p-value of the Welch t-test. Is this correct? Relatedly, sometimes the tables are referenced in the text but sometimes they are not. I would recommend referencing the tables consistently so readers know where to look to see results.\n\n(g) (Section 3) It would be nice to have a list of these metrics and what they mean for handy reference. I would suggest compiling one and adding it to an appendix.\n\n(h) (277) I am confused by this. Could you explain in greater detail or provide an example?\n\n(i) (Table 10) It would be interesting to see the average correlation for each of the categories described by the subsections in Section 3. For example, the average correlation for discourse feature metrics, syntax tree feature metrics, etc… this would give a nice sense of variation by category, which would help understand broad trends in the data.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for raising important and very detailed points to consider in our study, particularly with how the task can be framed, as well as possible additions to its limitations. We have taken note of all suggested improvements to our paper in terms of presentation, grammar, references, and content revisions. Please find our responses to your individual questions below. We hope that our responses are able to sufficiently address your questions, and we also hope that this may be reflected in updated scores.\n\n1. *\"The complexity metrics the authors use seem to be miscalibrated to their research question. The complexity metrics they use demonstrate complexity is not maintained for human responses. I would encourage the authors to find metrics that do not show such differences for humans, and then explore whether results are the same for models.\"* - This is a very good angle of discussion for the paper and we thank the reviewer for pointing this out. However, we cannot truly know what linguistic feature or metric would remain constant for humans and may or may not be for models unless we try all possible measures of text complexity in a generation experiment—thus, the whole methodology of the paper. \n\n2. *\"Given that the authors are reporting so many metrics, instead of the large number of tables, it would help to provide visualizations that can show the major trends in their data across metrics.\"* - Thank you for this recommendation.  Similar to Reviewer #1’s suggestion, we can provide some information on visualized direction by comparing the mean of each linguistic feature for the continuation whether they are lesser or greater than the prompt. We can do this by adding arrow lines (red for decrease and green for increase) for each column in the tables for our next revisions of the paper.\n\n3. *\"The authors report results from a model that was fine-tuned on the test set (GPT-DF). It would be better to report model behavior on data that it hasn’t seen during training.\"* - The data we used for finetuning GPT-DF are texts from the NEWSELA and ELG datasets, which are considered to be professionally-written. These datasets were not included in the generation procedure for Section 3, as the WritingPrompts dataset is the one used here.\n\n4. *\"The authors mention the possibility for this type of analysis (UCTG) to be the basis for an automatic evaluation metric. I think this paper would be much stronger if the authors followed through on this, and actually created some scripts to run the types of evaluations they report.\"* - This is a great point, and this is also the reason why we added Section 5 - Going Forward. We recommend that while researchers have the freedom to implement any linguistic metric using any tool or software, they would also have to consider factors such as taking into consideration user background and properties of the language they will work on. Our tone for this recommendation inclines more on being suggestive of the benefits rather than prescriptive. \n\n5. *\"(Line 28) Is this just true for story writing or for text generation in general?\"* - We believe that consistency still holds true even if applied broadly to text generation, but more strictly followed in story generation.\n\n6. *\"(Line 50) What’s the difference between text complexity and readability? Does it matter here?\"* - Text complexity and readability are often interchanged in previous works. However, we can further clarify this in our paper. We thank the reviewer for pointing this out.\n\n7. *\"(Line 192) A number of other decoding strategies have emerged in recent years, including locally typical sampling and truncation-sampling. I wonder if the results reported here generalize across decoding strategies?\"* - We would like our experiments to complement previous works that have also used the same dataset as WritingPrompts and generation settings (Fan et al., 2018; Delucia et al. 2021) for uniform comparison. We can add the choice of decoding method used in the Limitations section as a reference for researchers who want to explore this particular research direction.\n\n8. *\"It’s never stated what the numbers in the table are. I assume that they are the p-value of the Welch t-test. Is this correct?\"* - Yes, the values on the tables are the p-values of the Welch test with respect to the setup (columns) and linguistic feature tested (rows). We apologize for not being explicit about this in the discussion. We will revise Section 2 to improve the clarity.\n\n9. *\"(Line 277) I am confused by this. Could you explain in greater detail or provide an example?\"* - Previous works have supported that texts for lower levels of readability are usually short and simple. Thus, the use of POS features such as coordinating conjunctions, prepositional phrases, subordinating clauses, etc. may signal longer texts and would lead to increased complexity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(a) The authors have identified an interesting research topic that complements previous work.", "reasons_to_reject": "(a) The biggest reason to reject the paper is that the complexity metrics the authors use seem to be miscalibrated to their research question. The authors’ main intuition is that humans will maintain complexity between a text prompt and its continuation (an intuition that I share). However, the complexity metrics they use demonstrate complexity is not maintained for human responses. (That is, if I am reading the tables correctly; see (b) below.) This makes me think that their metrics are just too strong. I would encourage the authors to find metrics that do not show such differences for humans, and then explore whether results are the same for models. \n\n(b) There are a number of presentational issues with the paper. The biggest one has to do with the tables reported in Section 3. What are the numbers? I can guess from the text that they are p-values from the statistical tests, however I don’t believe that this is indicated in the text of the paper. In addition to the p-values of the tests, it would be very helpful to see the actual raw data that the authors are collecting, along with some estimate of the error. Finally, given that the authors are reporting so many metrics, instead of the large number of tables, it would help to provide visualizations that can show the major trends in their data across metrics.\n\n(c) Two comments about the models: First, the authors only test a GPT-2 model. Given that transformers are extremely easy to download and finetune these days, I would appreciate results from more models. Furthermore, if I am reading Section 2.2 correctly, the authors report results from a model that was fine-tuned on the test set (GPT-DF). It would be better to report model behavior on data that it hasn’t seen during training.\n\n(d) Finally, the authors mention the possibility for this type of analysis (UCTG) to be the basis for an automatic evaluation metric. I think this paper would be much stronger if the authors followed through on this, and actually created some scripts to run the types of evaluations they report. This would greatly increase the value of this research to the community!\n", "questions_to_authors": "(a) (28) Is this just true for story writing or for text generation in general?\n\n(b) (50) What’s the difference between text complexity and readability? Does it matter here?\n\n(c) I would mention in the intro that you actually provide tests to quantify whether uniformity is observed for humans too!\n\n(d) (171) One problem with this model is that you assume that humans do, in fact, produce uniformly complex material during generation. I think this assumption should be stated clearly. While I agree with the assumption, one problem is that the test you provide seem to indicate differently.\n\n(e) (192) A number of other decoding strategies have emerged in recent years, including locally typical sampling (Meister et al., 2022) and eta/trunctation-sampling (Hewitt et al., 2022). I wonder if the results reported here generalize across decoding strategies.\n\n(f) (Table 1) It’s never stated what the numbers in the table are. I assume that they are the p-value of the Welch t-test. Is this correct? Relatedly, sometimes the tables are referenced in the text but sometimes they are not. I would recommend referencing the tables consistently so readers know where to look to see results.\n\n(g) (Section 3) It would be nice to have a list of these metrics and what they mean for handy reference. I would suggest compiling one and adding it to an appendix.\n\n(h) (277) I am confused by this. Could you explain in greater detail or provide an example?\n\n(i) (Table 10) It would be interesting to see the average correlation for each of the categories described by the subsections in Section 3. For example, the average correlation for discourse feature metrics, syntax tree feature metrics, etc… this would give a nice sense of variation by category, which would help understand broad trends in the data.\n", "paper_topic_and_main_contributions": "Summary: This paper explores whether GPT-2 maintains uniform text complexity between the prompt and the output in a story-generating setting. The authors quantify text complexity (roughly equivalent text difficulty) via a slew of metrics, which have been used in the prior literature on text readability and text difficulty. They find that models, generally, do not maintain complexity between a prompt and output, however that various psycholinguistic features (such as age of acquisition) are correlated between prompt and output. Overall, I think that the authors’ main question is very interesting and well-framed. I have a couple of major concerns about the methodology, which I detail below. In addition, there are some serious presentational issues with the paper. Together, these make me hesitant to recommend this paper for acceptance. That being said, I strongly encourage the authors to go through another round of revisions and resubmit – I think they are on track towards a really promising paper!", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "(a) (158) I would recommend citing the transformers package from huggingface if this is, in fact, what you used.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "(a) (55) Output is consistent → output to be consistent\n\n(b) (89) that observes uniform → that observe uniform"}, "extra_text": {}}, {"review_id": "eSG9wWvDjd", "reviewer": "Reviewer_udN3", "summary": "", "questions": "- A: Has any other attempt been done on testing the complexity of the text generated by LLMs? If so, it would be advisable to add a Related Work section.\n- B: Line 145: can the same prompt have different continuations?\n- C: Line 468-69: you state \"highly-correlated features\". However, in Table 10, correlation is 0.188 at max. Is that an error? Plus, are they statistically significant? Please elaborate.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for raising important points that we should consider in our study. Please find our responses to your individual questions below. We have taken note of all suggested improvements to our paper in terms of presentation, grammar, references, and content revisions. We hope that our responses are able to sufficiently address your questions, and we also hope that this may be reflected in updated scores.\n\n1. *\"The assumption that a piece of text should be uniformly complex is not necessarily true. Several counter-example can be found: pedagogical texts, sub-stories in longer texts, changes of complexity. Part of this point is explained and mitigated in the limitations, however, I would like to see a bit more on this.\"* - This is a very good angle of discussion and was also raised by Reviewer #1. Indeed, longer texts may be analyzed in a finer-grained way to see the change in complexity from one part to another. However, our current work is currently limited to analyzing prompt-continuation pairs as a whole to be comparable with the methodology of previous works (De Lucia, et al 2021; Fan et al 2019). We thank the reviewer for pointing this out and we can include this in our limitations section in our next revision. We believe this finer-grained analysis of longer texts would be a promising next step for future research.\n\n2. *\"Has any other attempt been done on testing the complexity of the text generated by LLMs?\"* - Yes, there are a few works we cited throughout the paper such as the works of Roemmele et al (2017) and See et al (2019) on using a small set of 5-10 complexity metrics to evaluate GPT-2 model generations. In our work, we conducted a full linguistic investigation of 150+ features comparing generations from humans, base, and finetuned models.\n\n3. *\"(Line 145) Can the same prompt have different continuations?\"* - Some prompts from the WritingPrompts dataset used have more than one continuation. However, we resorted to using only one prompt is to one continuation (1:1) ratio for the sake of uniformity of processing. Likewise, our analysis is approximated across all features, so it is not necessary for one prompt to have multiple continuations.\n\n4. *\"(Line 468-69) You state \"highly-correlated features\". However, in Table 10, correlation is 0.188 at max. Is that an error? Plus, are they statistically significant? Please elaborate.\"* - What we meant by “highly-correlated features” is relative to the table contents. While a correlation value of 0.188 (not a typo) is not “high” by any standards in statistics, it is the highest value in the table. Nonetheless, we understand that this has confused the reviewer and will be revising \"highly-correlated features\" to “top-correlated features” instead. Most of the features are statistically significant at α < 0.05. We thank the reviewer for pointing this out and will revise accordingly to improve the clarity of discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work stems from an interesting idea, which is linguistically and cognitively motivated, introducing a novel task.\n- The paper is well-written and clearly organized.\n- The results are thoroughly reported.", "reasons_to_reject": "- The assumption that a piece of text should be uniformly complex is not necessarily true. Several counter-example can be found: pedagogical texts, sub-stories in longer texts, change of complexity for some particular effects etc. Part of this point is explained and mitigated in the limitations, however, I would like to see a bit more on this.\n- Only GPT models have been tested, partially limiting the generalizability of the results. However, I do believe that this can be left as a future work.\n- A manual analysis on texts that received particularly high and/or low scores would have been highly appreciated.", "questions_to_authors": "- A: Has any other attempt been done on testing the complexity of the text generated by LLMs? If so, it would be advisable to add a Related Work section.\n- B: Line 145: can the same prompt have different continuations?\n- C: Line 468-69: you state \"highly-correlated features\". However, in Table 10, correlation is 0.188 at max. Is that an error? Plus, are they statistically significant? Please elaborate.", "paper_topic_and_main_contributions": "This paper introduces a new task, Uniform Complexity for Text Generation (UCTG), aimed at studying whether generative language models are capable of generating continuations with the same level of complexity w.r.t. the given prompts. The authors test 176 features by comparing human prompts against continuations (i) written by humans and (ii) produced by several models from the GPT-2 family. The results show that both humans and GPT-2 continuations are inconsistent w.r.t. the complexity of the given prompts. However, they show that there are some similarities in the way humans and GPT-2 generate continuations.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 195: 0.70.\n- It would be advisable to put PDF or SVG figures.\n- Tables are not progressively presented in-text, e.g, Table 4 is before Table 3, 6 before 5, etc.\n- Could you add in one of the captions what the bold values in the tables stand for?"}}]

Author rebuttal:

Meta-review: Pros:
* The paper introduces a new task to study whether language models are capable of generating text with the similar level of complexity observed in the respective prompts. 
* All reviewers agree that studying the text complexity in prompted models is an interesting and novel idea, and will be of interest to the community.
* The authors present  a through analysis for the proposed task with a comprehensive set of complexity metrics.

Cons: 
* The authors needs to improve their motivation and the limitations of the study elaborating on 1) the assumption that "humans will maintain complexity between a text prompt and its continuation" (given that the complexity metrics tell a different story), 2) the fact that a long text could have varying level of complexities (in contrast this work focuses on prompt continuation only), and 3) whether or not a more fine grained text analysis would be more informative. 
* All reviewers pointed out that the analysis in the paper will improve with larger set of models. Currently, only GPT-2 results are analysed. 
* The paper could also improve with better presentation and visualisation of their results. The authors have responded to these requests and have agreed on improving these aspects.
Decision: Accept-Findings