On the Automatic Generation and Simplification of Children's Stories
Authors: Maria Valentini
Jennifer Weber
Jesus Salcido
Téa Wright
Eliana Colunga
Katharina von der Wense
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Generation
NLP for Education
LLMs

Abstract: With recent advances in large language models (LLMs), the concept of automatically generating children’s educational materials has become increasingly realistic. Working toward the goal of age-appropriate simplicity in generated educational texts, we first examine the ability of several popular LLMs to generate stories with properly adjusted lexical and readability levels. We find that, in spite of the growing capabilities of LLMs, they do not yet possess the ability to limit their vocabulary to levels appropriate for younger age groups. As a second experiment, we explore the ability of state-of-the-art lexical simplification models to generalize to the domain of children’s stories and, thus, create an efficient pipeline for their automatic generation. In order to test these models, we develop a dataset of child-directed lexical simplification instances, with examples taken from the LLM-generated stories in our first experiment. We find that, while the strongest-performing current lexical simplification models do not perform as well on material designed for children due to their reliance on large language models behind the scenes, some models that still achieve fairly strong results on general data can mimic or even improve their performance on children-directed data with proper fine-tuning, which we conduct using our newly created child-directed simplification dataset.

On the Automatic Generation and Simplification of Children's Stories
Maria Valentini
Jennifer Weber
Jesus Salcido
Téa Wright
Eliana Colunga Katharina von der Wense" v
University of Colorado Boulder
Johannes Gutenberg University Mainz
(first. last)t@ooaocccoo edu
Abstract
Target randumly from pre filtered dataset
Prompt from of age-bnsed prompts
With recent advances large language models (LLMs). the concept of automatically gener- ating children's educational materials has be- come increasingly realistic. Working toward the goal of age-appropriate simplicity in gen- erated educational texts, we first examine ability of several popular LLMs generate ries with properly adjusted lexical and readabil- ity levels. We find that, in spite of the growing capabilities of LLMs, they do not yet possess the ability to limit their vocabulary to levels ap- propriate for younger age groups. experiment, we explore the ability of state-of- the-art lexical simplification models to general- ize the domain of children's stories and, thus, create an efficient pipeline for their automatic generation. order to test these models, we de- velop a dataset of child-directed lexical simplifi- cation instances, with examples taken from the LLM-generater stories in our first experiment. We find that, while the strongest-perfiee pernesst-erfforiig lexical simplification models do not perform as well on material designed for children due to their reliance on LLMs, model that performs well on general data strongly improves its per- formance milldren-directed data with proper finetuning, which we conduct using our newly created child-directed simplification dataset.
Large language model
Lexical simplification model
Age-appropriate and individualized vecabulary-enrichinng story
second
Figure 1: Simple example of a pipeline for the genera- tion of educational children's stories.
As we will describe in more detail in Section 2, children's s stories are important for both entertain- ment and education. Automatic story generation for children increases their potential for broader impact by making possible personalize sto- ries, making them increasingly relevant for each individual child. As one example, tailoring stories to a specific child's interests could allow them to become more easily interested in reading, which could improve their literacy skills. Another pos- sible application could be the teaching of specific target words to preschoolers via stories. In this paper, will keep the latter use case in mind.
1 Introduction
Large language models (LLMs), such as GPT-3 or ChatGPT, are able to produce stories that are far more coherent and fluent than stories generated by state-of of-the-art models from even a couple of years ago, such as GraphPlan (Chen et al., 202 la) and Hoin-and-Write (Yao et al., 2018). However, most of the already limited work automatic story generation focuses on stories for an adult audience. Children's 's stories are not frequently topic of in- terest, despite how crucial carly literacy to future success (Walker et al., 1994).
While, at first look, stories that have been gener- ated by LLMs seem (and generally are) better than previous attempts, has not been systematically evaluated if their children's 's stories adhere to what one would expect from the genre. In this paper, focus on the simplicity of generated stories measured by the age of acquisition of their indi- vidual words. Specifically, we assume that we are interested in generating stories teach words with an age of acquisition (AoA) from 9 to children who are preschool-agee -aged (around 2-5). Including
Formerly: Katharina Kann

---

other complex words or concepts, however, can make story-based vocabulary learning more diffi- cult, as they detract attention from target words and complicate the ascertainment of context clues. Thus, we want to include the target words in the stories, but we do not want any other words to have an AoA > 6.
simplification models trained on adult data are suit- able to simplify automatically generated children's s stories only when properly finetuned, i.e., the best- performing models for adult text do not generalize well without additional training, but fintuning is effective for our domain, even with limited data.
To summarize, our contributions are follows: (I) We examine and compare the ability of sev- eral LLMs to adjust for age-appropriate levels of readability and lcxical simplicity in automatically generated and individualized educational stories. (2) We explore how effectively state-of-the-ar lexi- cal simplification models perform in the domain of children's 's stories in order to supplement the inabil- ity of LLMs sufficiently adapt their vocabularies. (3) We provide a public dataset for the simplifica- tion of child-directed text in order to promote the advancement of models for this purpose.
For the first experiment conducted in this paper, found in Section 4, use 3 different LLMs for generation and ask the following research ques- tions: (RQ1) What is the simplicity of generated stories for different LLMs? (RQ2) How do dif- ferent prompts, with different descriptions of the target age group, influence story simplicity for dif- ferent models? We find that, in spite of LLMs' growing abilities for story generation, they strug- gle to control for age-appropriate simplicity; in comparison our dataset of numan-generated sto- ries for the same demographic, the models' sto- ries exhibit scores that are over 17% worse for all readability metrics tested (Flesch Reading Ease, Flesch- Level, Gunning-Fog Index, and Au- tomated Readability Index).
Background: Why Should LLMs Generate Children's Stories?
In child development, small early differences can compound into big long-term effects. One example of this is the relationship between early vocabu- lary size, literacy, and later academic achievement. Early vocabulary size strongly related to reading ability in 2nd and 3rd grade (Walker et al., 1994; Fewell and Deutscher, 2004), and even when con- trolling for vocabulary size in Kindergarten, read- ing ability in 4th grade is associated with vocabu- lary growth through 10th grade (Duff et al., 2015). Although there is a recognition of this relationship. large gaps in vocabulary size persist into elemen- tary school and beyond: e.g., Biemiller and Slonim (2001) estimate the vocabulary of 4th graders in the lowest quartile to be less than half the size that of 4th graders in the highest quartile, and similarly sized gap is found in empirically-basee estimates throughout adulthood (Brysbaert et al., 2016).
Motivated by those findings, in Section 5, we then turn towards simplification models. Simpli- fication models have mostly" been developed for text directed adults. Here, we investigate two leaders of the 2SAR-2022 Shared Task on Multilin- gual Lexical Simplification (Saggion et al., 2023) in English, UniHD (Aumiller and Gertz, 2022) and MMU (Vásquez-Rodrííuue et al.. 2022), with regards to their ability to generalize to the chal- lenging out-of-domail setting of children's 's stories. For this, generate new dataset of annotated instances for lexical simplification, using age of ac- quisition as metric for identifying complex words and pulling examples from our newly created cor- pus of LLM--genrated stories for human annotators to simplify. We then use this dataset to evaluate the models' performance on automatically gener- ated children's 's stories. Our experiments show that, while UniHD performs considerably worse, achiev- ing an accuracy of only 30.52% in comparison to the 42.89% accuracy achieves on the TSAR-EN dataset for general lexical simplification, the ordi- narily weaker UofM& MMU model is actually able to perform better on the hhild-iirected dataset when finetuned, achieving an accuracy of 47.37% com- pared to its 28.95% on TSAR. We conclude that
Many vocabulary enrichment programs based on shared reading with a caregiver have been devel- oped to try to address this gap, with mixed success. Early vocabulary interventions are generally based on storybook reading with a parent or teacher. A meta-analysis focusing on vocabulary intervention studies children in pre-K K and K concluded that although such interventions may increase oral lan- guage skills, they are not powerful enough to close the gap, even when implemented at this early age (Marulis and Neuman, 2010). Experts agree that intensive, individual- interventions would be
'Exceptions exist but are relatively old and use since- deprecated tochnology, such as (De Belder and Moens, 2010- 01-01) and (Vu et al., 2014).

---

necessary make a difference but acknowledge that the infrastructure investment required for some- thing on that scale would be substantial (Suskind et al., 2013).
setting up scaffolding for the story ahead of time in order to ensure the coherence of the eventual generation model. With LLMs such as ChatGPT (OpenAI, 2022) and Bard (Google, 2022), however, systems for automatic story generation can begin to rely less on this strategy of planning story structure ahead of generation.
With recent improvements in LLMs, the concept of using natural language processing techniques to automate the child-by-chill customization of edu- cational materials has become increasingly realistic. In particular, state-of-the- the-rrt models have proven to be effective for generating individualized reading materials without the labor cost of having a human modify them by hand, making this process more practical in lower-resource settings facing educa- tional discrepancies. Developing efficient pipelines for personalized ation is one way ito address this so called "achieve- ment gap" in early childhood education, providing opportunities for individualized education where they otherwise may not be available.
More modern approaches that implement LLMs such Future Sight (Zimmerman et al., 2022) and MEGATRON -NNTLL (Xu et al., 2020) allow for more creative modification during and after gener- ation. Worderaft (Yuan et al., 2022), for instance, allows end users to collaborate with OpenAI's GPT- 3 in order to continually modify stories throughout the process of their generation.
eyaabulary-enrichhng story gener-
One facet of story generation that has shown es- pecially prevalent applications of late has been the generation of children's 's stories. Early childhood literacy has proven to be a significant indicator of a child's future academic success (Walker et al., 1994), so emphasizing generation for childhood literary advancement is one way in which mod- em NLP techniques can be especially impactful. Though many story generation systems have as- pects that can likely be transferred to children's stories, few studies intentionally focus their gener- ation systems directly children.
There are several challenges that should be ad- dressed in order to make the automatic generation of individualized educational children's 's stories a more fcasible idea. One primary concern is ensur- ing that the stories simple enough to be under- stood by the target demographic. Stories which are too complex or which contain too many unknown words could make understanding the meanings of their target words based on the context provided more difficult. Additionally, studies show that hav- ing fewer unfamiliar detractors allows children to focus on and retain target words more effectively (Horst, 2013). As such, it is important that auto- matic story generation models be able to control for the simplicity and readability of the stories outside of the words they intend to teach. Because of the potential personalized stories have for increasing both specific educational potential and overall en- gagement in reading. this paper focuses on how LLMs and lexical simplification models can be used for this purpose.
3.2 Prompt-tuning
A topic that has seen increasing amounts of atten- tion as L.LMs have become more and more popular is prompt-tuning, or the study of how the modifica- tion of prompts to LLMs affects their output. The concept of prompt-tuning in the sense in which it is used in this paper was originally proposed in Lester et al. (2021), who propose the idea of modifying prompts for GPT-3 in way that af- fects the model's results quantifiably, similar how parameter tuning affects ordinary machine learning models. Other papers such as Hu et al. (2022) and Gu et al. (2022) reaffirm the effective- ness of prompt-tuning for LLMs, demonstrating that it can have an even more significant effect than traditional parameter finetuning. This paper does not place large emphasis on prompt-tuning, but we do perform prompt-tuning on a smaller scale investigate how specifying different target groups affects the simplicity of generated stories.
3 Related Work
3.1 Automatic Story Generation
The automatic generation of stories is a task which has seen vast changes in its methodology with the recent advent of LLMs, requiring increasingly less human intervention to mimic human-written sto- ries. Even fairly recent automatic story generation models such as that by Li et al. (2013), GraphPlan (Chen et al., 2021b), and the Plan-and- -nr-iie-- system (Yao et al., 2019) focused much their effort on
3.3 Lexical Simplification
Lexical simplification describes the process of iden- tifying words which are too complex for some

---

cost and has target demographic and replacing them with syn- onyms which are easier to understand. Lexical simplification (LS) has been a commonly studied NLP tasks for several years, and early models such as De Belder and Moens (2010-01-01) and Vu et al. (2014) highlighted in particular the applications of lexical simplification for children or for non-native speakers (Petersen and Ostendorf, 2007). More recent models such LSBert (Qiang al., 2020), which was created by finetuning the BERT masked language model on the task of lexical simplification, focus instead on the task of general LS, bringing up the question of whether these more modern, higher- performing models can be generalized to work as effectively on children-directed text.
per 1000 tokens.
ChatGPT ChatGPT, or GPT-3355 is an- other model created by OpenAl using reinforce- ment learning from human feedback, released in November 2022 (OpenAI, 2022). Unlike Instruct- GPT, ChatGPT finetuned in supervised setting by using human-created AT assistant dialogue sam- ples, making it more equipped for dialogue. is currently free to use as a part of its research pre- view, but its code is not yet publicly available.
Vicuna Vicuna is a LLM created by finetun- ing the open-source LLaMa (Meta, 2023) on the ShareGPT dataset of user conversations. Accord- ing to preliminary evaluations done using OpenAl's GPT-4, Vicuna is able to achieve 92% of the perfor- mance of ChatGPT (Chiang al., 2023), Vicuna has the advantage over both GPT-3 and ChatGPT, however, that it can be used publicly available code. It is included in this study to test the capabilities of openly available LLMs to generate age-appropriate educational stories for preschoolers. This experiment uses the version of Vicuna with 7 billion parameters, which is built off of LLaMa's 7 billion parameter model.
Even more recently, the TSAR-2022 Shared Task on Multilingual Lexical Simplification (Sag- gion et al., 2023) has drawn more attention to the improvement of LS models such as the winning UniHD model (Aumiller and Gertz, 2022). This shared task also led to the development of the ConLS system (Sheang et al., 2023), which is modified version of LSBert created after the TSAR shared task and obtains state-of-the of-the-art results the TSAR-2022 dataset.
4 Experiment 1: Assessing Readability
4.2 Datasets
Age of Acquisition Data The dataset used to identify words in the model-generated stories that should be simplified in order to reduce their com- plexity and increase their educational potential is the English Lexicon Project's Age of Acquisition dataset (Kuperman et al., 2012). This dataset con- sists of over 31,000 words along with the estimated average age at which they are learned based on crowd-sourced data collected by researchers at six universities.
First, we investigate the readability of automati- cally generated stories for 1) different models and a variety of prompts. We generate total of 250 stories for each model: 50 per model-prompt combination.
4.1 Models
InstructGPT InstructGPT is group of GPT-3 models finetuned via reinforcement learning from human feedback. Trained 1.3 billion parameters, it released by OpenAl in January 2022 as part of its series of generative pre-trained transformer (GPT) models. These models use data gathered by crawling the internet to predict how a series text tokens should be completed (Brown et al., 2020). The InstructGPT series of models are unique from other GPT-3 models in their intentional alignment with their purpose, which is completing text given natural language instruction, opposed the in- herent misalignment faced by models that just aim predict statistically what word(s) should come next (Ouyang et al., 2022). Specifically, this ex- periment implements OpenAl's Text-DaVinei-003 model, which can be used at the cost of $0.0200
Books for Preschoolers In order have set of stories with which compare the ones generated by the above-described LLMs,, we use the Books for Preschoolers dataset (BfP) from Wiemerslage et al. (2022). It consists of 1026 human-written sto- ries intended for children ages 2-5. Of all the words in this corpus, $8.61% % can be found in the Age of Acquisition dataset (Kuperman et al., 2012) which is used to test the simplicity of the stories gener- ated by the LLMs. Among the words that could be found in the AoA data, 83.08% were below the age threshold we compare in stories, which is The stories included in BiP are commercially availaale, professionally-writte

---

picture books (i.e., books that have illustrations in every single page) intended to be read to preschool- ers or by early readers, such as Good Night, Moon and The Very Hungry Caterpillar. Transcribed sto- ries the corpus contain an average of 52 sen- tences and 9.4 words per sentence. The authors themselves or the publishers designated the stories' age range that qualifies them for inclusion in this dataset.
4.4 Metries
Currentl, automatic readability metrics are limited and largely consist of ones that are significantly out- dated (Vajjala, 2022). These measures, although well-established and widely used, are coarse over- simplifications of language use. Since are using several different measures consistently between the human- and computer-writtn stories, however, we believe these imperfect measures serve as good starting point for comparison against the BfP cor- pus. As such, to judge the simplicity of the stories generated in Experiment 1, we use the following metries.
4.3 Prompts
With our prompts, aim to provide the model with the necessary information for generation of stories around the target words. In addition, we want to encourage simplicity, i.e., stories that are easily understandable by young children. We as- sume our target group consists of children aged 6 or younger.
Average Age of Acquisition We go through the stories generated by each model and find each word's age of acquisition. We then take the av- erage from all of these words so we can judge the ability of each model to simplify their lexicon to reflect that of their target demographic.
We experiment with the following prompts:
Write story for preschooler containing the following words: w1, w2, w3, w4, w5
Average Highest Age of Acquisition We check each story generated by model and find the word in it that has the highest age of acquisition. We then take the average of these scores for each model judge each model's ability to avoid using words in stories which are too complex for their target demographic.
Write story for 3-year-old containing the following words: w1, w2, w3, w4, w5
Write a story for a 4-year-old containing the following words: w1, w2, w3, w4, w5
Write story for y-aarroddd containing the following words: w1, w2, w3, w4, w5
Readability Scores: Flesch Reading Ease We use readability scores to judge the relative ease of reading each of the models' stories. The Flesch Reading Ease score is calculated using the follow- ing formula: Reading Ease 206.835 (1.015 x Average Sentence Length) (84.6 x Average Sylla- bles per Word).
Write a children's story containing the fol- lowing words: w1, w2, w3, w4, w5
Target Words With the target demographic of preschool-aged children in mind, we select the target words for our -enereeeerteed stories from words in the AoA dataset with ages of acquisition between 6 and 9. Starting with all the words in this age range, we go through specific filtering process that includes steps such removing ad- verbs and words tagged with more than one part of speech, avoiding multiple words derived from the same lemma, and removing words with missing or low concreteness scores. The target words are then reviewed by multiple annotators and scored in three categories: learnability, imageability, and appropriateness. The guidelines relating to these categories (as they were presented to the annota- tors) are included in Appendix A. At this point, only the highest scoring words in each category are kept, resulting in a list of 150 nouns, 50 verbs, and 50 adjectives. The complete list of target words can also be found in Appendix A.
Readability Scores: Flesch-Kincale Grade Level Another readability metric we use to test the diffi- culty of reading cach story is the Kincaid Grade Level, which is computed via the following formula: Kincaid Grade Level (0.39 x Average Sentence Length) (11.8 Average Sylla- bles per Word) 15.59.
Readability Scores: Gunning-Fog Index An- other readability metric we use is the Gunning-Fog Index, calculated as follows: Gunning-Fou Grade Level 0.4 (Average Sentence Length Per- centage of Hard Words), where "hard words" are defined words with three more syllables that are not (i) proper nouns, (ii) combinations of easy words or hyphenated words, or (iii) two-syllable verbs made into three with -ss and edd endings.

---

Model
Prompt
Average AoA
Highest AoA
% Valid
% Appropriate
Preschool
4.75
9.15
90%
0%
3-year-old
4.71
8.94
94%
0%
InstructGPT
4-year-old
9,21
94%
0%
5-year-old
4.69
9.2
92%
0%
child
9.67
96%
0%
Preschool
4.74
9.7
54%
0%
3-year-old
4,64
9.57
58%
0%
Vieuna
4-year-old
4.69
8.95
50%
0%
5-year-old
4,66
9.34
52%
0%
child
4.76
10.23
48%
0%
Preschool
4.62
8,94
94%
0%
3-year-old
4.64
9.01
96%
0%
ChatGPT
4-year-old
4.63
8.76
94%
0%
5-year-old
4.67
9.24
98%
0%
child
4.74
9.62
98%
0%
BiP
4.6
8.87
100 %
4.78%
Table 1: Full Experiment results by model and prompt used. Highest AoA refers average highest age of acquisition. % Valid refers to the percent of stories including all desired target words, and % Appropriate is the percent of stories containing only words with AoA S The human-written BfP corpus is included for comparison.
Readability Scores: Automated Readability In- dex The final readability metric is the Automated Readability Index, whose formula is:
Readability Metrics
Flesch Reading
Flesch-Kincaid level
characters, ) 0.5
words
1.71 X
sentences 21.13
words
% of Valid Stories We further look at the of valid stories to judge the models' ability to adhere to the prompts assigned. Stories are considered invalid if they are missing one or more of the as- signed target words.
Gunning-Fog Index
Automated Readability
% of Age-Appropriate Stories Last, we com- pute the % of age- age-appropriate stories to judge the models' ability to adhere to the specified age group. Stories are considered invalid if they contain words with an age of acquisition higher than 6.
2-
InstucisFTT
Vieuna
ChaiGPT
B/P
Figure 2: Readability metries for each model, well
the Books for Preschoolers human-writ written dataset (BfP). Note that the direction of the arrow the side of each chart indicates the direction of improved readability.
4.5 Results
Some key results for first experiment shown in Table 1. After running the above-described ex- periment, we find that although LLMs are generally able to simplify the average word difficulty their stories to age-appropriate levels, they are unable to avoid including some words with age of acquisition levels significantly higher than their target demo- graphic. Further, we find that none of the 750 total generated stories stayed within the age range of 6 or younger. Though it is common for some children's stories contain words that are more complex, others refrain from using such words in order to cater to their target demographic (c.g., 49 stories in the Books for Preschoolers dataset). Having this
ability is a key difference between computer- and human- human-generated stories, highlighting an area which our automatically generated stories could use improvement.
In terms of readability, the stories generated by models score significantly worse than those in the BfP dataset. While the average Flesch Read- ing Ease (FRE) in the BfP dataset is approximately 89.37 (out of 100), the average FRE among the
stories generated by the models is only 74.22. Con- trarily, for the Flesch- Flesch-Kincaid Level (FKL) and

---

Gunning-Fog Index (GFT), a lower score indicates body of text being easier to read. In the FKL and GFI metrics, the BfP stories score on average 2.9 and 5.23, respectively. Among the stories gener- ated by the LLMs, meanwhile, these scores average out to 6.44 and 8.87. Full results of this readability analysis can be seen in Figure 2. With regard to our second research question, we find that age-specific prompt-tuning has little effect on the simplicity of children's ss stories generated by LLMs.
dataset, or CDS for short.
5.2 Models
We use two of the three best systems of the TSAR- 2022 Shared Task on Multilingual Lexical Simpli- fication to test the ability of lexical simplification models to generalize to the domain of children's stories: UniHD (Aumiller and Gertz, 2022) and UofM&MMU (Vásquez-Rodriquezz et al., 2022). In addition to examining the ability of statc-of-the- art lexical simplification models to simplify our computer-generated children's stories, we also ex- periment with LLMs for the task to see if they outperform any models designed specifically for simplification.
5 Experiment 2: Story Simplification
The results shown in Section 4.5 demonstrate serious need for improvement concerning age- appropriate simplicity in automatically generated children's 's stories. Thus, in our second experiment, we examine to what extent current state-of ootte--o-the-rrr the-art lexical simplification models generalize to the do- main of children's s stories, as well as how this could be beneficial in the automatic generation of educa- tional children's is stories by LLMs.
UniHD's model is created using an en- semble of six different configurations/prompt com- binations from GPT-3. Its results are generated by caleulating aggregate ranking of the outputs of its different GPT-3 configurations and prompts. It demonstrates satte-of-the-see performance the area of lexical simplification, with an accuracy score over 25% higher than LSBert (Qiang et al., 2020), which is regarded as one of the most popular and effective LS models to date.
UniHD
5.1 Dataset Creation
To the best of our knowledge, there are currently no datasets for lexical simplification focused on children. As such, the first step in our second ex- periment is to create such a dataset. We are able to do this by using the corpus of children's 's stories created in Section 4, identifying complex words using the Age of Acquisition dataset, and having human annotators identify simpler synonyms for these complex words (in cases where such syn- onyms exist). In the annotation process, each in- stance consists of the sentence from which the com- plex words was taken, as well as the complex word itself. Annotators are tasked with finding a sim- pler synonym for each instance that could replace the complex word in the sentence without chang- ing the sentence's 's meaning. Each instance is then reviewed by two additional annotators, profes- sor and PhD student in NLP, to ensure that the meaning of the sentence is retained. Only instances deemed to be valid are kept, meaning all annota- tors agree the sentence's 's meaning was unchanged and the newly suggested synonym has a lower age of acquisition than the original word. In total, annotate 750 instances randomfy selected from our corpus of LLM--gnnrrted stories. After filtering out instances deemed to be invalid, our final dataset consists of 3 355 simplification examples. We refer to this dataset as our Child- Direeced Simplification
MM While UolM&MMU's performs considerably lower on accuracy on the TSAR dataset than UniHD does, it can be finetuned with additional data. Based on the BERT masked language model, the UofM&MMU model goes through three distinet steps in its simplification process. The first involves candidate generation based on different prompt templates be provided to BERT. The second finetunes BERT and subse- quently ranks and selects candidates. Finally, the candidates are post- pett-pocessed in order to filter out noise and remove any antonyms that may appear. For this model, we are able to finetume using a train- ing set consisting of 70% of the CDS dataset. The remaining 30% is used as a test set.
Vicuna, ChatGPT, InstructGPT We further ex- periment with the three LLMSs used in our Experi- ment 1. We use the following set of prompts to get data for our full range of metrics:
Name a simpler synonym that could replace the word [word] the following sentence: [sentence]
"The complete https//giihah.ccm/maaaaasallCCl
datasct
can
be
found
at

---

Name two simpler synonyms that could re- place the word [word] in the following sen- tence: [sentence]
dataset, predicting the top human-selected substi- tution with 47.37% accuracy. This demonstrates that finetuning can result in ordinary lexical simpli- fication models being able to generalize to simplify child-directed text and that even better results could be achieved if better-performing models allow for the same level of finetuning.
Name three simpler synonyms that could re- place the word [word] in the following sen- tence: [sentence]
We conclude that, while LLMs and LLM-based models struggle to simplify children's 's stories, mod- cls which allow for finetuning on domain-speciffe data can perform as well as or even better on chil- dren's 's stories than they do on adult-directed cor- pora. In terms of our overall pipeline, we conclude that it is in fact plausible to generate and simplify children's 's stories using LLMs for generation and finetuned lexical simplification models to simplify overly complex words.
5.3 Metries
For Experiment 2, we use three metrics measure the performance of the simplification models.
Accuracy We define accuracy the ratio of in- stances in which the top-ranked candidate gener- ated by the model is equal to the top synonym chosen by human annotators.
Simplification Validity This metric represents the ratio of instances where the model chooses a top candidate with a lower age of acquisition than the original word.
Conclusion
In this paper, we investigate the ability of several current LLMs to generate age-appropriately sim- plified stories for children, as well as an examina- tion of how modern lexical simplification models generalize to the domain of children's stories to enhance their educational potential. We demon- strate that, in spite of their growing capabilities. modern LLMs are unable generate children's 's stories with age-appropriate simplicity, particularly in comparison to their human-written counterparts. Because of these shortcomings found in the auto- matically generated stories, our second experiment (Section 5) focuses on whether or not ordinary lex- ical simplification models generalize to the domain of children's stories, due the lack of current LS models that focus on children-directed corpora. We find that some models which are ordinarily lower- performing than their LLM--poered counterparts have the potential to perform well in the domain of simplifying child-directed text, when properly finetuned.
Accuracy @k This is the ratio of instances in which at least one of the top-k candidates generated by the model is equal to the top synonym chosen by human annotators. We calculate this with = 2 and k = 3.
5.4 Results
Full results of our second experiment with regard to the tested simplification models can be found in Table 2. Upon running the UniHD model on our child-directed simplification dataset, find that the performance of the model is significantly worse than is on the TSAR shared task English dataset. In terms of accuracy, the model is able to generate a top candidate equal to the one selected by human annotators 30.52% in comparison to 42.89% in the adult-directed dataset. Regarding a pure LM em- ployed with prompting, find that, with accuracy scores lower than all but one of the other models and validity scores significantly lower than any of them, LLMs are not effective tools for this task.
Over the course of experiments, we fur- ther create a corpus of vocabulary-driven LLM- generated children's stories as well as an anno- tated lexical simplification dataset, CDS, intended specifically for the domain of children's 's text and using examples taken from this above-mentions automatically generated stories. We provide these datasets publicly in order to promote further search in this area.
However, we find different results for the UUUM&MMU model in combination with finetun- ing. On the original TSAR-EN dataset, UniHD outperforms UMUM&MMU by over 15% accuracy. Without finetuning, the UoMM&MU model also performs significantly worse the child-directed dataset, with an accuracy score of just 8.42%. After being finetuned with a portion the CDS dataset, however, the model is able to score even better than the best perforringn model does on the TSAR
In future work, we hope to further improve the automatic generation of customized children's s sto-

---

TSAR-EN
Accuracy
Validity
Accuracy@2
Accuracy
UniTID
0.429
0.381
0,611
0.686
U
0.290
0.370
0,453
0.531
0.290
0.467
0,453
0.531
CDS
Accuracy
Validity
Accuracy@2
Accuracy
UniHD
0.305
0.616
0.368
0.379
UOTM&MMU
0.084
0.716
0.147
0.189
0.474
0,937
0.537
0.600
InstructGPT
0.111
0.394
0.238
0.248
ChatGPT
0.263
0.610
0.378
0.422
Vicuna
0.140
0.438
0.219
0.254
Table 2: Full results Experiment 2. Comparison of the UniHD and UMfM&MMU models' performance on the adult-directed dataset (TSAR-EN) and all tested models' performance on our dataset (CDS). Validity refers to the model predictions' simplification validity for each dataset.
ries by adding models for other tasks to our genera- tion pipeline, such as one that can detect coherence errors or one that can improve readability.
Acknowledgments
We thank the anonymous reviewers for their help- ful comments. This research was supported by the NSF under grant IIS 2223917. The opinions expressed are those of the authors and do not repre- sent views of the NSF.
Limitations
We use a limited number of LMs and simplifica- tion models in our experiments. and the number of prompts we explore is also rather small. Thus, while our experiments feature state-of-the-arr mod- els, cannot exclude with absolute certainty that other models prompts might lead different results. Our dataset is small as well, and it could be improved with the help of more annotators. Future research could include the use of more workers to create a significantly larger dataset, potentially through the use of gamification for data collection.
References
Dennis Aumiller and Michael Gertz. 2022. UniHD at TSAR-2022 shared task: Is compute all we need for lexical simplification? In Proceedings of the Work- shop on Text Simplification, Accessibility, and Read- ability (TSAR-2022), pages 251-258, Abu Dhabi, United Arab Emirates (Virtual). Association for Com- putational Linguistics.
Andrew Biemiller and Naomi Slonim. 2001. Estimating root word vocabulary growth in normative and advan- taged populations: Evidence for common sequence of vocabulary acquisition. Journal of Educational Psychology, 93:498-520.
Ethics Statement
Regarding the ethical considerations for this study, we find that the harms are minimal due to the relatively confined nature of the experiments; all annotations were performed voluntarily and with consent. Potential benefits of this study include the advancement research in the area of child- directed lexical simplification and improvements in efficiency for the creation of personalized edu- cational material for young children. Though the results of this research are eventually intended for the demographic of children, vulnerable popu- lations were involved in this study up this point. If automatically generated stories given read to children, important to verify in advance that they are safe for the target population, as current models cannot guarantee this.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam Mc- Candlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot leam- ets. (arXiv:2005.144 14165). ArXiv:2005. 14165 [es].
Marc Brysbaert, Michaël Stevens, Pawel Mandera, and Emmanuel Keuleers, 2016. How many words do we know? practical estimates vocabulary size depen- dent on word definition, the degree of language input

---

and the participant's age. Frontiers in Psychology. 7:1116.
Loren M. Marulis and Susan B. Neuman. 2010. The ef- fects of vocabulary intervention on young children's word learning: A meta-analysis. Review of Educa- rional Research, 8003):300-335
Hong Chen, Raphael Shu, Hiroya Takamura, and Hideki Nakayama. 2021a. Graphplam: Story generation by planning with event graph. CoRR, 102//200.00977.
Meta. 2023. Llama.
Hong Chen, Raphael Shu, Hiroya Takamura, Hideki Nakayama. 2021b. Graphplan: Story generation by planning with event graph. (arXiv:2102.0297)) ArXiv:2102.02977 [es].
OpenAL. 2022. Chatgpt.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow in- structions with human feedback. Stti::2203002555: ArXiv:2203.02155 [cs].
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Jan Belder and Marie-Francine Moens. 2010-01-01. Text simplification for children. Proceedings of the SIGIR workshop accessible search systems, pages 19 26.
Sarah E. Petersen and Mari Ostendorf. 2007. Text sim- plification for language learners: a corpus analysis. In Slate.
Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, and Xindong Wu. 2020. Lsbert: A simple frame- work for lexical simplification. (tXiv:2006.14939) ArXiv:2006.11 14939 [cs].
Dawna Duff. Bruce Tomblin, and Hugh Catts. 2015, The influence reading on vocabulary growth: case for a matthew effect. Journal of Speech, Lan- guage, and Hearing Research, 58(3):853-86 864.
Horacio Saggion, Sanja Štajner, Daniel Ferres, Kim Cheng Sheang, Matthew Shardlow, Kai North, and Marcos Zampieri. 2023. Findings of the tsar- 2022 shared task on multilingual lexical simplifica- tion. (arXiv:2002.02888) ArXiv:2302.00 02888 [cs].
Rebecca R Fewell and Barbara Deutscher. 2004. Con- tributions of early language and maternal facilitation variables to later language and reading abilities. Jour- nal of Early Intervention, 26(2):132-145
Google. 2022. Bard.
Kim Cheng Sheang, Daniel Ferrés, and Horacio Sag- gion. 2023. Controllable lexical simplification for english. NurXiv:2302.02900 ArXiv:2302.02900 [cs].
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. Ppt: Pre-trained prompt tuning for few-shot learning. StxXiv2210.043322 ArXiv:2109.0433 093323 [es].
Dana Suskind, Patricia Kuhl, Kristin Leffel, Susan Landry, Flavio Cunha, and Kathryn M Neckerman. 2013. Bridging the early language gap: A plan for scaling up. Bridging the Thirty-Million Word Gap.
Jessica Horst. 2013. Context and repetition in word learning. Frontiers in Psychology. 4(149).
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning: Incorpo- rating knowledge into prompt verbalizer for text clas- sification. (arXiv:2108.Am .0.0335... ArXiv:2108.0200 [cs].
Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment re- search. (arXiv:2105.00973) ArXiv:2105.00973 [cs].
Laura Rodruez-Rodriguez, Nhung Nguyen, Matthew Shardlow, and Sophia Ananiadou. 2022. UoM&MMU at TSAR-2022 shared task: Prompt learning for lexical simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022) pages 218-224, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguisties.
Victor Kuperman, Hans Stadthagen-Gonzalez,, and Mare Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior Research Meth- ods, 44(4):978-999 9900.
Brian Lester. Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning. (arXiv:2104.08691) ArXiv:2104.08691 [es].
Tu Thanh Vu, Giang Binh Tran, and Son Bao Pham. 2014. Learning to simplify children stories with limited data. In Intelligent Information and Database Systems, Lecture Notes Computer Science, page 31-41, Cham. Springer International Publishing.
Boyang Li, Stephen Lec-Urban, George Johnston, and Mark Riedl. 2013. Story generation with crowd- sourced plot graphs. volume 27, page 598-604.

---

Dale Walker, Charles Greenwood. Betty Hart, and Ju- dith Carta. 1994. Prediction of school outcomes based on early language production and socioeco- nomic factors. Child Development, 65(2):006-221.
Rate each word with respect to their imageability. High imageability (5) words will easily evoke a mental image in your mind; LOW imageability (1) words will evoke a mental image with difficulty or not at all.
IMAGEABILITY
Adam Wiemerslage, Miikka Silfverberg, Changbing Yang, Arya McCarthy, Garreu Nicolai, Ellinaa Col- unga, and Katharina Kann. 2022. Morphological processing of low-resource languages: Where we are and what's next. In Findings of the Association for Computational Linguistics: ACL 2022. pages 988- 1007, Dublin, Ireland. Association for Computational Linguistics.
A.2 Full List of Target Words
accordion, acrobat, almond, anteater, antelope, anthill, bakery, bandanna, banjo, beagle, billboard, blizzard, bobcat, bookmark, bookshelf, bouquet. bugle, campground, cashew, catcher, cavern, chan- delier, cheetah, chef, chimpanzee, clipboard, cob- web, collie, comet, confetti, cookbook, cottage, cowbell, crater, crescent, cricket, cyclist, denim, desert, dome, doorknocker, dumbbell, earmuff, earplug, earring, easel, eggplant, elk, ferryboat, fireplace, flute, forearm, fountain, gator, glacier, gnome, golf, grove, gutter, hairnet, hammock, head- stand, hedgehog, hexagon, hiker, hourglass, ice- berg, iguana, island, jaguar, jersey, kayak, kiwi, lantern, lifeboat, limousine, llama, lobster, locker, macaw, mansion, maze, microwave, mole, moss, mountaintop, museum, musician, newt, nostril, or- chard, pelican, petunia, pinwheel, piranha, platy- pus, pompom, poncho, propeller, receipt, rink, rocker, sardine, sax, sequin, shrimp, shrub, sib- ling, skillet, skylight, skyscraper, sloth, snowshoe, songbird, sparrow, spatula, speck, spotlight, squid, stadium, stairwell, statue, stethoscope, stopwatch, suitcase, tangerine, taxicab, tennis, thimble, thun- derstorm, tightrope, tongs, tortilla, toucan, trol- ley, trombone, trouser, tuba, tulip, tumbleweed, tusk, tutu, vase, violet, violin, visor, volleyball, warthog, wreath, xylophone, bald, bearded, beige, blond, blurry, breakable, bubbly, bushy, chalky, chilly, cloudless, crumbly, electric, feathery, flo- ral, foamy, foggy, frosty, gooey, grassy, greenish, hatless, hilly, lilac, longhaired, lumpy, magenta, moonless, moonlit, mossy, plaid, prickly, puffy, reddish, seaside, sleeveless, slimy, smoky, starry, stormy, stretchy, sunless, sunlit, swampy, thorny, turquoise, undersea, wintry, wooded, wrinkly, ap- plaud, awaken, bulldoze, curl, dangle, darken, dec- orate, deflate, deliver, dine, dotted, drove, enlarge, erupt, exhale, expand, fetch, flatten, halve, hover, illustrate, inflate, invite. juggle knit, magnify, masked, mimic, mow, munch, perform, recline, re- paint, rotate, serve, sew, skydive, sniff, soak, squint, stumble, sunbathe, topple, unfold, unhook, unlock, unpack, unroll, unzip
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. Megatron-entrl: Controllable story generation with external knowledge using large-scale language models. (a0X0v:0010.00840) ArXiw:2010.00840 [cs].
Lili Yao. Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. (arXiv:1811.05700) ArXiv:1811.0571 [es].
Lili Yao, Nanyun Peng. Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2018. Plan-and- write: Towards better automatic storytelling. CoRR, abs/1811.0570..
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ip- polito. 2022. Wordcraft: Story writing with large language models. In 27th International Conference on Intelligent User Interfaces, IUI '2, page 841 1-852, New York, NY, USA. Association for Computing Ma- chinery.
Brian D. Zimmerman, Gaurav Sahu, and Olga Vech- tomova. 2022. Future sight: Dynamic story generation with large pretrained language models. (arXiv:2212.09947)) ArXiv:2212.099477 [cs].
A Appendix
A.1 Annotator Guidelines
Rate each word with re- spect to how appropriate they are for children. A HIGH uppropriateness (5) word will be totally fine for preschooler; LOW appropriateness (1) word should NOT appear in a story for preschoolers.
APPROPRIATENESS
LEARNABILITY Rate each word based on how likely you think preschooler is to be able to learn it from story. HIGH learnability (5) words should be easy to learn from story; LOW learnability(1) (1) words would be nearly impossible to learn from a story.


Reviews:
[{"review_id": "noaPRNDdpQ", "reviewer": "Reviewer_Si5t", "summary": "", "questions": "- In the use case presented at pages 2-3, it is not clear for me who will read the text to children aged 2 to 6? It does not seem that most of them will be able to read the stories by themselves, so to which extent your system would be different to the tests carried out with caregivers reading the story? In my opinion, I do not see a very large market for personalized stories for children of this age, but I might be wrong. \n- Could you please define the notions of learnability, imageability, and appropriatness as they were explained to the annotators?\n- It is not clear for me how the % of Age-Appropriate Stories were assessed. How many experts did you used? Were they able to reach a good agreement?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review! We think you characterize the main contributions of our paper very well, and we appreciate the thought and detail you put into reviewing our work. We also found your reasons to reject insightful and would like to address those here as well as in our paper in order to ameliorate any concerns.\n\nFor one, you mention that our methodology includes readability metrics which are outdated and only capture aspects of the text such as word and sentence length, which we agree is not very conducive to the aims of our paper. The readability measures we used, although well-established and widely used, are coarse oversimplification of language use. We appreciate the inclusion of (Vajjala, 2021) for reference and have examined it for possible improvements to our current measurements. This paper mentions the lack of readily available tools or code as one of the main limitations to automatic readability assessment, which is something our own research has reflected as well. It cites just two studies that have code publicly available (Ambatti et al., 2016 and Howcroft & Demberg, 2017), one of which is for directly comparing readability between two samples as opposed to providing a metric for readability in general, and both of which can only assess readability at an individual sentence level. Since we are using several different measures consistently between the human- and computer-written stories, however, we believe these imperfect measures serve as a good starting point for comparison against the BfP corpus, which is a corpus of commercially available published picture books for children ages 5 and younger. \n\nFurther, we also agree that our annotation process lacks description in this iteration of the paper and would like to include full definitions of learnability, imageability, and appropriateness as they were presented to our annotators in the appendix of our paper for future readers. Thank you for pointing this out! In regard to the percentage of age-appropriate stories, this was simply decided automatically by the stories’ words’ age of acquisition. We also would like to clarify that for each example to be added to our newly created CDS dataset, the top answer had to be agreed upon by 3 or more annotators, two of whom always included a professor and PhD student working in the field of Natural Language Processing.\n\nYou also mention some questions regarding the intended use case for these stories. You are correct, the children that are the targets for these stories are preschoolers and are not independent readers. Thus, they will require an adult caregiver to read the stories to them. Caregivers reading stories to children, often called “book sharing”, is an important way in which children increase their vocabulary. Finding ways to increase a child’s vocabulary by including target words in contexts that appeal to the child’s current interests is one of the long-term goals of this project.\n\nFinally, thank you for all of the additional sources you have provided! These are great resources surrounding dyslexic children in relation to our topic and definitely will enable more exploration in that direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Three real contributions in a field that will become more and more important.\n- Good paper, well-written and reporting solid work.", "reasons_to_reject": "- The authors seems to consider that De Belder and Moens (2010) are deprecated technologies for LS, but they base their analysis of the text reading difficulty in readability formulae dating back more than 70 years, ignoring advances in the field (see Vajjala, 2021). In particular, these formulas measures the length of word rather than AoA or word frequency, which might not be the best choice in relation to paper’s aims. \n- The different annotation processes are described in a rather naive way, with no sign of annotation guidelines, not computation of interrater agreements. \n\nVajjala, S. (2021). Trends, limitations and open challenges in automatic readability assessment research. arXiv preprint arXiv:2105.00973.", "questions_to_authors": "- In the use case presented at pages 2-3, it is not clear for me who will read the text to children aged 2 to 6? It does not seem that most of them will be able to read the stories by themselves, so to which extent your system would be different to the tests carried out with caregivers reading the story? In my opinion, I do not see a very large market for personalized stories for children of this age, but I might be wrong. \n- Could you please define the notions of learnability, imageability, and appropriatness as they were explained to the annotators?\n- It is not clear for me how the % of Age-Appropriate Stories were assessed. How many experts did you used? Were they able to reach a good agreement?", "paper_topic_and_main_contributions": "This article tackles two related research questions. On the one hand, it aims to determine whether LLMs are capable of generating stories for young children (2 to 6 years old) in which the age of acquisition of words and the readability of these stories is appropriate. On the other hand, they investigate the use of Lexical Simplification (LS) to compensate for the weaknesses of LLMs at the lexical level. The authors found that three LLMs (including ChatGPT) are not able to generate children stories at an appropriate level of difficulty, which is an interesting result to dampen the current trend to believe that “every task should be done prompting an LLM”. In addition, they apply state-of-the-art LS model to the generated texts with the aim of simplifying lexical load of these stories. They show that a model that can be fine-tuned performs better than an LLM-based model for children stories. Finally, a third contribution consists in a dataset of 315 sentences in which a complex word (as regards AoA) has been simplified by experts.    \n\nIn my opinion, these three contributions are valuable and fit for EMNLP. Automatic generation of stories of different difficulty levels is also an area that is bound to flourish over the next few years, and this article is a good milestone. In addition, the paper reads well, is based on a use case that is well-motivated with theoretical work from various fields, and reports experiments based on a solid methodology. There are, however, a few shortcomings that prevent me from being completely enthusiastic about this paper.   \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- For resources and ATS work relative to children, there are some papers focusing on dyslexic children, such as Rello et al., 2014 or Gala et al., 2020.\nGala, N., Tack, A., Javourey-Drevet, L., François, T., & Ziegler, J. C. (2020). Alector: A parallel corpus of simplified French texts with alignments of misreadings by poor and dyslexic readers. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 1353-1361).\nRello, L., Baeza-Yates, R., & Llisterri, J. (2014). Dyslist: An annotated resource of dyslexic errors. In Conference on Language Resources and Evaluation LREC-2014, page 1289–1296, Reykjavick, Island.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Several references are incomplete, in the bibliography. Please check them systematically.\n\nl.152: Biemiller and Slomin (Biemiller and Slomin, 2001) → Biemiller and Slomin (2001)\nl.578: which the at least → “the” should be suppressed.\nl.628: models are unable generate →  models are unable to generate"}}, {"review_id": "soK9t7aiAu", "reviewer": "Reviewer_Psov", "summary": "", "questions": "1.\n\nThe 'Books for preschoolers' (BFP) corpus is cited for  Wiemerslage et al., 2022.\nHowever, that paper barely mentions it on two lines and gives no details at all (!!!).\nThe present manuscript provides very scant detail about this corpus, all it says is:\n\n\"It consists of 1026 human-written stories intended for children ages 2-5. Of all the words in this corpus, 88.61% can be found in the Age of Acquisition dataset\"\n\nSince this collection is central to the reported work, more details should be provided about the corpus. \nWho wrote the stories? (professional writers?)  How long are the texts on average (word counts)?  Who was qualified to designate them to ages range 2-5? \nAre those commercial publications or something else (e.g. free materials)?\n\nWhy almost 12% of words were not found in the AoA data?  - what kinds of words did not have AoA: exclamations? proper names? etc.\nCan you provide percentages by such groups (e.g. proper names)?\n\nNote: since the BFP corpus is not released and is not public, this can affect the reproducibility of the study,\nHowever, I did not lower the reproducibility score for that (as releasing such data can be difficult). \n\n2.\n\nThe use of the Flesch Readability Ease (FRE) is somewhat problematic. For FRE, the scale starts with 100 (and goes down), but 100 corresponds to US grade level 5  (see  https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests),\nwhich corresponds to ages 10-11, well above your targeted level of ages 2-5 or even 6-9. So, this measure does not have enough resolution or sensitivity to measure readability for the age levels of interest.\n\nMoreover, the paper says (lines 475-478) that the average FRE in the BFP corpus is 89.37 (well below 100), which is at about beginning of 6th grade level, i.e. age 10-11.\nDoes that imply that the BFP corpus is actually not suitable for children ages 2-5?!!!  Or maybe the BFP corpus is suitable, and FRE is just an inadequate measure for this age?  Something is incongruous here. \n\n3.\n\nSection 5.1. Human annotators selected synonyms.  Only in the Limitations section we get to know that those annotators were crowd-sourced. More details need to be provided about the adequacy of those annotators to the task. \n\n4.\n\nThere is some potential to be misleading.\nIn the Conclusions section, lines 639-645, the text says \"...LLM...have the potential to perform very well in...\".\nWhen one reads that formulation (without the numbers), and most readers read only abstracts and conclusions, one might consider that LLMs perform well on this task.\nBut actually the better result is just 36% (line 603), which is nice for research, but is way far from 'well performing' for any applied use. \n\n \n\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for this thorough review of our paper! We appreciate the consideration put into your questions for us and think that adding these details into the paper would be a great way to improve it for publication. \n\nIn your first question regarding the BfP dataset, we agree that it is not well described in our current submission of the paper. We would like to clarify that the stories included in BfP are commercially-available, professionally-written picture books (i.e., books that have illustrations in every single page) intended to be read to preschoolers or by early readers, such as *Good Night, Moon* and *The Very Hungry Caterpillar*. Transcribed stories in the corpus contain an average of 52 sentences and 9.4 words per sentence. The authors themselves or the publishers designated the stories’ age range that qualified them for inclusion in this dataset.\n\nRegarding the words in the BfP corpus that were missing from the Age of Acquisition data, only 30.02% of those missing could be found in the English dictionary, meaning 69.98% were either names, made up words, or other non-word entities. Of those that were found in the dictionary, the large majority were either proper nouns (e.g., February, Arizona, American), plurals of nouns whose singular forms are in the AoA data, or past tenses of verbs whose present forms are in the AoA data.\n\nYour statement addressing the age mismatch with the readability metrics we are using (specifically FRE) is also an excellent point. Given that the books contained in the corpus are all commercially available, professionally written stories published for this age group, we believe that using the imperfect readability measures across the board and in comparison to the BfP corpus as a control is a reasonable place to start. Thanks to your comment, however, and upon further examination of readability formulae, we were able to find a measure that has sensitivity for younger ages to test as well; the Automated Readability Index is designed for ages 5 and up, meaning it can more accurately test whether stories are fit for our 2-5 age range. We found similar results using this metric, with the BfP corpus scoring an average of 1.67 (around Kindergarten level) and our model-generated stories scoring an average of 5.72 (around 4th grade level). Full results are shown below.\n\n| **BfP (Baseline)** | **InstructGPT** | **ChatGPT** | **Vicuna** |\n|--------------|--------------|----------------|----------------|\n| 1.67         | 5.60         | 5.46           | 6.10           |\n\nWe also agree that our current manuscript is lacking in details relating to the annotations used for our study. As we mentioned to Reviewer 1, we would like to clarify that for each example to be added to the newly created CDS dataset, the top answer had to be agreed upon by 3 or more annotators, two of which always included a professor and PhD student working in the field of Natural Language Processing.\n\nFinally, thank you for your comment on the wording of our conclusion; we agree that this could potentially be misleading and would definitely prefer to present as accurate a picture of our results as possible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall the study is very interesting and even exciting. Really nice!\n\nAs the popularity of novel LLMs and their wide adoption are growing, the aspect of text complexity can be very important for educational uses.\nAs  study 1 shows, LLMs do not perform adequately, at least for  very young children.\nAs study 2 shows, simplification for very young children is improving, but is not quite solved yet.\n\nThe idea of using AoA as a measure for vocabulary is a good one. \n\nThe study with simplification is actually a different direction from study 1, but it is conceptually well-related to study 1, \nso overall the paper is very good.", "reasons_to_reject": "There are some inadequacies with some of the measures. But no reason to reject.", "questions_to_authors": "1.\n\nThe 'Books for preschoolers' (BFP) corpus is cited for  Wiemerslage et al., 2022.\nHowever, that paper barely mentions it on two lines and gives no details at all (!!!).\nThe present manuscript provides very scant detail about this corpus, all it says is:\n\n\"It consists of 1026 human-written stories intended for children ages 2-5. Of all the words in this corpus, 88.61% can be found in the Age of Acquisition dataset\"\n\nSince this collection is central to the reported work, more details should be provided about the corpus. \nWho wrote the stories? (professional writers?)  How long are the texts on average (word counts)?  Who was qualified to designate them to ages range 2-5? \nAre those commercial publications or something else (e.g. free materials)?\n\nWhy almost 12% of words were not found in the AoA data?  - what kinds of words did not have AoA: exclamations? proper names? etc.\nCan you provide percentages by such groups (e.g. proper names)?\n\nNote: since the BFP corpus is not released and is not public, this can affect the reproducibility of the study,\nHowever, I did not lower the reproducibility score for that (as releasing such data can be difficult). \n\n2.\n\nThe use of the Flesch Readability Ease (FRE) is somewhat problematic. For FRE, the scale starts with 100 (and goes down), but 100 corresponds to US grade level 5  (see  https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests),\nwhich corresponds to ages 10-11, well above your targeted level of ages 2-5 or even 6-9. So, this measure does not have enough resolution or sensitivity to measure readability for the age levels of interest.\n\nMoreover, the paper says (lines 475-478) that the average FRE in the BFP corpus is 89.37 (well below 100), which is at about beginning of 6th grade level, i.e. age 10-11.\nDoes that imply that the BFP corpus is actually not suitable for children ages 2-5?!!!  Or maybe the BFP corpus is suitable, and FRE is just an inadequate measure for this age?  Something is incongruous here. \n\n3.\n\nSection 5.1. Human annotators selected synonyms.  Only in the Limitations section we get to know that those annotators were crowd-sourced. More details need to be provided about the adequacy of those annotators to the task. \n\n4.\n\nThere is some potential to be misleading.\nIn the Conclusions section, lines 639-645, the text says \"...LLM...have the potential to perform very well in...\".\nWhen one reads that formulation (without the numbers), and most readers read only abstracts and conclusions, one might consider that LLMs perform well on this task.\nBut actually the better result is just 36% (line 603), which is nice for research, but is way far from 'well performing' for any applied use. \n\n \n\n\n", "paper_topic_and_main_contributions": "This paper examines the capability of LLMs to generate stories at the appropriate age level (stories for age of about 5).\nThe reported work includes two studies.\nOne study focused on different ways of prompting LLMs  and measuring the vocabulary complexity of resulting outputs relative to a database of Age of Acquisition  (AoA) (vocabulary in English).\nThe results were also evaluated for age-appropriateness with classic readability formulae.\nThe second study looked at the ability of some recent lexical-simplification systems to simplify stories to the level required.\n\n ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 490: \"metric shown in 1\" - did you mean table 1?", "Justification_for_Ethical_Concerns": "I don't see any ethical concerns for this work."}, "extra_text": {"Missing_References": "1.\n\nAoA data (section 4.2) has the wrong reference. It cites Balota et al. 2007, (line 351 and again line 363). However that dataset does not have AoA,\nit has lexical decision and naming latencies.\nThe dataset with 31K AoA ratings is probably this one: \n\nVictor Kuperman & Hans Stadthagen-Gonzalez & Marc Brysbaert\n  \nAge-of-acquisition ratings for 30,000 English words\n\nDOI 10.3758/s13428-012-0210-4\n\n\n\n "}}, {"review_id": "yGBuEVxwLS", "reviewer": "Reviewer_6sAM", "summary": "", "questions": "Why did you select target words in the range of 6-9 for the age of acquisition when some of prompts mention younger ages (i.e. “write a story for a [3,4,5]-year-old”)?\n\nAlso, if you using this 6-9 range, why is a story considered invalid if it contains a word with an age of acquisition higher than 6?\n\nRegarding the results in Table 1, how do the human-authored BfP stories compare with these numbers?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate this review, and you bring up several good points that we certainly agree would improve the overall quality of the paper. \n\nYour mention of our second experiment lacking results regarding the use of an LLM for simplification is an excellent point. We did a fair amount of experimentation with LLMs before deciding we needed to explore other models, so it would definitely make sense to formalize this and include it in the paper to further demonstrate the soundness of our conclusions. As such, we present below the experiment we have now run, using several different prompt templates to test the models’ performance in the same ways as the other LS models. The prompts used were as follows:\n\n1. *Name a simpler synonym that could replace the word [word] in the following sentence: [sentence]*\n\n2. *Name two simpler synonyms that could replace the word [word] in the following sentence: [sentence]*\n\n3. *Name three simpler synonyms that could replace the word [word] in the following sentence: [sentence]*\n\nThese prompts were selected in order to get all metrics for comparison (accuracy@1-3 and validity). The table below shows the results of this experiment.\n\n| **Model**       | **Accuracy@1** | **Validity** | **Accuracy@2** | **Accuracy@3** |\n|-----------------|--------------|--------------|----------------|----------------|\n| **InstructGPT** | 0.111        | 0.394        | 0.238         | 0.248        |\n| **ChatGPT**     | 0.263        | 0.610        | 0.378         | 0.422        |\n| **Vicuna**         | 0.140        | 0.438        | 0.219         | 0.254        |\n\nWith accuracy scores lower than all but one of the other models we tested and validity scores significantly lower than any of them, the results from even the best of these language models demonstrate that LLMs are not effective tools for this task.\n\nWe also tried another line of full story prompting but found it was also not effective for our purposes. Even with our most detailed prompt:\n\n*Simplify the following story to improve its readability for preschool-aged children:*\n*[story]*\n*Try to use simpler words and not to shorten the story.*\n*However, do not remove the following words: [targets]*\n\nthe LLMs were only able to generate valid simplifications that left all target words in the story less than 60% of the time, despite only decreasing the stories’ average age of acquisition by just over 0.02 years.\n\nAdditionally, your suggestion of including more details to demonstrate the importance of the CDS dataset is a good one, and we would add these details into the next iteration of the paper. In regard to the specific questions you asked, we would first like to clarify that the dataset is composed of 315 individual simplification examples, each consisting of the word in need of simplification, the sentence the word comes from for context, and the simpler synonym that was chosen by annotators to replace it. These examples were selected randomly from sentences in the computer-generated stories that were flagged as containing one or more words above our target age of acquisition. We intentionally use the already fairly simple computer-generated stories in order to test how well models can simplify for young children, i.e. simplifying words that are already simple enough to not be replaced in adult-directed lexical simplification. Our hope is that having this domain-specific simplification dataset available will encourage future research in this area and improve upon current methods.\n\nIn your questions section, you ask why we selected words with age of acquisitions ranging from 6 to 9. Since the goal is for these stories to support vocabulary growth, we wanted to select target words that most children in our target age range would not know. The AoA metric is just an estimate, and individually, children can learn vocabulary words with an AoA above their actual age as long as they encounter the words in their environment and they are presented accompanied by the scaffolding context of language that they do understand, so we want the rest of the story to be written in the style and at a level for our target age range. We also reference (Horst, 2013) in our background section, which mentions that a story containing other unknown elements aside from the novel target words negatively impacts children’s learning of these words. Therefore, if any other words in our stories (all language surrounding our 5 target words with higher ages of acquisition) had a higher age of acquisition as well, we considered the story to be too complex for the child and thus invalid. \n\nWe also appreciate your suggestion for an additional paper to examine (Eldan & Li, 2023) and agree it has some valuable information closely related to our paper. The goal for the GPT-generated stories is quite similar to ours in that the model is instructed to refrain from using words that are too complex for a young child to understand. It seems that even with the more precise instruction demonstrated in this paper, however, the model still fails to contain its vocabulary to the desired age range. Using the smaller validation set to estimate the statistics of the overall TinyStories dataset, we find that the average age of acquisition for the stories is 4.46, and the average highest age of acquisition is 8.29. These do show slight improvement over ours, which have an average AoA of 4.66 and an average highest AoA of 9.54, but still highlights the need for further simplification in our use case.\n\nFinally, thank you for pointing out the lack of comparison with the human-written stories in Table 1. We did not include BfP in this table due to the fact that it only had one statistic as opposed to the five prompt-based statistics that are available for the model-written stories. We would like to modify our Figure 2, which compares the model-written stories with the human-written ones in BfP, to include this missing information on age of acquisition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The overall applied objective of the work is well-motivated. The authors effectively argue for the practical benefit of the task: being able to quickly generate children’s stories, especially ones that can be customized to a particular child’s interests, could improve students’ reading ability.\n\nThis paper also benefits from defining the generation task through specific objective criteria: in particular, the authors use an existing resource (the English Lexicon Project’s Age of Acquisition dataset) to determine the lexical constraints for the stories. This enables the generated stories to be readily evaluated in terms of how well they respect these constraints.", "reasons_to_reject": "The paper claims that its two-stage pipelined approach, where an LLM generates the initial story and a separate lexical simplification (LS) model is applied to this story, is effective for the task. While the paper demos this pipeline, the experiments do not show that this particular approach is favorable to any alternative approach. In particular, the paper does not compare its pipeline to an LLM-only approach or an LS-only approach, nor does it compare the LLMs and LS models to each other. These two components occupy different sections in the paper, and in each section different datasets and evaluation metrics are used. The paper assumes that the LLMs and LS models will receive different inputs: for LLMs, it’s an instruction prompt, while for the LS models, it’s an existing story. However, an LLM could similarly take an existing story as input (along with a prompt instructing the model to simplify it), which would enable it to be directly compared to the LS models. This should be included in the experiment to strengthen the paper's claims that LLMs are not sufficient for the task.\n\nThe paper presents a new dataset for the task, CDS, but its design and purpose is not adequately detailed. The paper utilizes an existing dataset of children’s stories, BfP. The LLM experiments utilize BfP for evaluation, while the LS experiments use CDS. More clarity is needed about the added value of CDS in relation to BfP. Is it that CDS explicitly annotates the lexical mapping between a story and its simplified version? What stories are used as the original source for CDS? Is it the human-authored BfP stories (as implied by Lines 509-511), or is it the LLM-generated stories (as implied by Lines 528-529)? If the task is to generate a simplified version of a complex-language story, then neither the BfP nor LLM stories seem well-suited to represent the “complex” side of the stories in a dataset for this task. This doubt could be resolved with some statistical description of CDS in terms of the frequency of words identified as complex in the original stories and thus replaced in the simplified versions. Line 531 indicates there are “315 simplification examples”, but it’s not clear whether an example is a single word or an entire story, and how many words per story are changed.", "questions_to_authors": "Why did you select target words in the range of 6-9 for the age of acquisition when some of prompts mention younger ages (i.e. “write a story for a [3,4,5]-year-old”)?\n\nAlso, if you using this 6-9 range, why is a story considered invalid if it contains a word with an age of acquisition higher than 6?\n\nRegarding the results in Table 1, how do the human-authored BfP stories compare with these numbers?", "paper_topic_and_main_contributions": "This paper presents a system for generating children’s stories, where the stories must meet certain vocabulary constraints based on the age-of-acquisition of words. The work examines whether selected LLMs produce stories that meet these constraints when prompted with instructions to do so. It finds that these particular LLMs tend to produce words that are above the desired age of acquisition for children’s stories, thus failing to meet the task requirements. To address this, the authors examine models that have been shown to be useful for the specific task of lexical simplification. They apply these models to the LLM-generated outputs to yield some lexical changes that satisfy the age-of-acquisition constraints.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Even though it may not be peer-reviewed yet, a very relevant paper to consider citing is this one:\n\nEldan and Li (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2 refers to the human-authored stories as a “baseline”, which is not the best term since “baseline” typically refers to a weak model. Renaming this to simply “human” would be clearer. \n\nEven though it’s explained in the text, it may be helpful in Figure 2 to add an up or down arrow to each of the three x-axis captions to indicate whether lower or higher scores are more desirable, since this differs between the three metrics."}}]

Author rebuttal:

Meta-review: The reviewers agree that this is an interesting and well-motivated study, and while there were some concerns raised about the lack of certain comparisons/evaluations and a lack of detail about the annotation process, discussion with the authors helped resolve these areas of concern, for the most part. They presented some follow up results and discussion, which should be included in the next draft of the paper (particularly the experiments regarding the different stages of the pipelined approach).
Decision: Accept-Main