Readability ≠ Learnability: Rethinking the Role of Simplicity in Training Small Language Models
Authors: Ivan Lee
Taylor Berg-Kirkpatrick
Venue: COLM 2025 (2025)
Keywords: child-directed language
developmentally inspired data
small language models
TL;DR: This paper argues statistical simplicity (low n-gram diversity), not human readability, is the critical factor enabling coherence emergence in small language models trained on synthetic datasets like TinyStories.

Abstract: Recent studies suggest that very small language models (SLMs) can generate surprisingly coherent text when trained on simplified, child-directed corpora such as TinyStories. These findings have been interpreted as evidence that readability—characterized by accessible vocabulary, familiar narrative structure, and simple syntax—plays a key role in enabling such capabilities to emerge. In this paper, we challenge that interpretation. We construct synthetic datasets with matched structure but varied readability, and find that readability alone does not predict coherence or learning efficiency in SLMs. Models trained on complex, adult-level text perform comparably to those trained on simplified language, and even exhibit faster development of coherence during training. Instead, we show that statistical simplicity, as measured by n-gram diversity, is a stronger predictor of learnability. Our findings caution against the growing trend of anthropomorphizing language model training—drawing parallels to human cognitive development without empirical basis—and argue for more precise reasoning about what properties actually support capability emergence in small models.

Published as a conference paper at COLM 2025
Readability * Learnability: Rethinking the Role of Simplicity in Training Small Language Models
Ivan Lee & Taylor Berg-Kitee Kirkpatrikakkk UC San Diego (iylee, tberg) @ucsd. edu
Abstract
Recent studies suggest that very small language models (SLMs) can gener- ate surprisingly coherent text when trained on simplified, child-directed cor- pora such as TinyStories. These findings have been interpreted as evidence that readability- charattrddd by accessible vocabulary, familiar narrative structure, and simple syntax-play a key role in enabling such capabilities to emerge. In this paper, challenge that interpretation. We construct synthetic datasets with matched structure but varied readability, and find that readability alone does not predict coherence learning efficiency in SLMs. Models trained on complex, adult-level text perform comparably to those trained on simplified language, and even exhibit faster development of coherence during training. Instead, we show that statistical simplicity, as measured by n-gram diversity, is a stronger predictor of learnability, Our findings caution against the growing trend of anthropomorphizing lan- guage model wraning-drawwng drawing parallels to human cognitive development without empirical basis-and argue for more precise reasoning about what properties actually support capability emergence in small models.!
Introduction
Recent studies have shown that very small language models (SLMs) can generate surpris- ingly coherent text when trained on the TinyStories dataset-a synthetic corpus of short, child-directed narratives written in highly readable language (Eldan & Li, 2023). These findings have led researchers to draw a connection between the simplicity of the dataset and the capabilities of the resulting models, with some suggesting that the use of simplified, developmentally appropriate language may play a key role in enabling coherence at small scales (Haga et al., 2024; Muckatira et al., 2024).
But what exactly is responsible for the success of TinyStories? Is it the readability of the text-short sentences, common words, familiar structure-thal enables these models to succeed? Or are the benefits better explained by other properties of the data, such as its synthetic origin, its low lexical and structural diversity, or the statistical regularities that result from its template-based generation process? While the TinyStories authors themselves do not make strong causal claims, the broader community has often cited the dataset in ways that imply such a connection. In practice, TinyStories is frequently described corpus of simple children's 's stories, and its effectiveness is often linked, implicitly or explicitly, to that simplicity (Theodoropoulos et al., 2024; Bunzeck & Zarrieß, 2023).
The term "simple" in this context is ambiguous. On one hand, it may refer readabil- ity-that is, how easily a text can be understood by a human reader, particularly child. On the other hand, it may refer to statistical simplicity: low entropy, high redundancy, and a narrow distribution over token sequences, often measurable through metrics like n-gram diversity. These are distinct notions, and conflating them risks misunderstanding what actually enables small models to generalize.
httpss/s/ignnnnaeeeco/o/lllllles 6dad1 1 13c4c3eal

---

Published as a conference paper at COLM 2025
In this paper, we disentangle these two notions of simplicity. We construct synthetic datasets with matched structure but varied readability, using prompt templates with identical struc- ture but explicitly differing in intended readership (child vs. adult) to ensure consistent format and narrative framing. We then train small transformer models on these datasets and evaluate their ability to generate coherent text. We find that models trained on complex, adult-level language perform comparably to those trained on simplified, child-directed text-and in some cases, coherence emerges earlier during training. These findings suggest that readability is not the key factor enabling coherence in SLMs.
These results challenge the intuition that child-directed language plays special role in en- abling language models to generalize. While the TinyStories authors do not make this claim explicitly, the dataset is often framed-by citations and surrounding discourse- asseeaasse devel- opmentally inspired intervention (Edman & Bylinina, 2023; Yam & Paek, 2024; Feng et al., 2024). This framing aligns with a broader and growing trend: the anthropomorphization of language models. As Shanahan (2023) and Placani (2024) argue, anthropomorphization can be more than a harmless shorthand-it risks distorting scientific reasoning, misrepre- senting model capabilities, and shaping misguided intuitions about how language models learn. In this case, attributing coherence to readability conflates human developmental simplicity with statistical learnability, obscuring what properties actually drive capability emergence in small models. Our findings instead suggest that statistical simplicity- ratherr than readability or developmental relevance-is a stronger predictor of learnability in SLMs.
2 Dataset Construction
Our investigation hinges on comparing models trained on datasets with specific prop- erties. Therefore, we began by constructing datasets guided by three desiderata: (1) Controlled Readability: Our primary goal is isolate readability differences between batasets-speciffe
differentiating child child-directed versus adult-directed language-to better understand how readability affects model coherence, while holding other properties constant. (2) Statistical Simplicity: We aim to minimize variance in statistical complex- ity-operationalized through metrics such as n-gram diversity-across datasets that differ in readability or thematic domain, as our primary hypothesis is that statistical simplicity significantly influences the learnability of SLMs. (3) Consistent Quality: We strive to maintain uniformly high dataset quality, ensuring variations in readability domain do not substantially degrade text quality, thus allowing us to attribute differences in model performance specifically to readability or statistical simplicity. These terms are defined and quantitatively validated in Section 3.
Guided by these desiderata, and ensuring fair comparisons by standardizing dataset size to approximately 1 billion tokens (for both synthetic and sampled corpora), we construct or select the following datasets for comparison:
TinyStories The TinyStories dataset consists of synthetic children's stories generated via structured, prompt-based pipeline using proprietary language models. Specifically, the authors created prompts from a fixed template, introducing variability by randomly sampling vocabulary items from a curated set of child-friendly words and narrative features such as dialogues or plot twists. While this approach is intended to promote high readability and narrative diversity, its reliance on proprietary models limits reproducibility.
LlamaTaaee-JJr We reconstruct the data generation pipeline introduced by TinyStories, using open-weight models 1aama-3.1-8B-Instteee
This produces a reproducible approxi- mation of TinyStories. We employ the same fixed prompt template, curated child-friendll vocabulary, and narrative features as the original TinyStories dataset, aiming to maintain consistency in readability, statistical simplicity, and quality. See Figure A20 for the template.
GalesGGRE We adapt the Llama pipeline to target adult readers by modifying the fixed prompt template to explicitly instruct the use of vocabulary intended for college- educated adults. Specifically, we replace the curated child-friendly vocabulary with more sophisticated GRE-level vocabulary. Narrative structures and feature distributions (e-g.,
2

---

Published as a conference paper at COLM 2025
inclusion of dialogue, twist endings) remain identical, intending to isolate readability the primary differentiating factor. The specific template is detailed in Figure A21.
Domain Variants We further extend the LlamaTaale--RRE pipeline by adapting the fixed prompt template to create three additional synthetic datasets, each targeting a distinct thematic domain while maintaining the same GRE-level vocabulary. These datasets include SaaaaTTTe--isttryy (short summaries of historical events), Saaaaalee-Sport (fictional sports articles), and LlamaTales- nalism). These domain variants are intended to help assess the robustness and generaliz- ability of our findings regarding readability and statistical simplicity across varied content domains. Prompt templates for these variants are shown in Figures A22, A23, A24.
(news-style narratives resembling mainstream jour-
Standard Pretraining Data We complement our synthetic datasets with subsets from established real-world pretraining datasets, primarily as points of reference that represent substantially higher statistical complexity. These datasets include our primary dataset, FineWeb-Edu (Lozhkov et al., 2024) highly filtered web dataset previously demonstrated to effectively train performant language models-an
supplementary datasets such as Sliml'jaam (Soboleva et al., 2023) and Dolma (Soldaini et al., 2024). While we do not attempt to directly generalize our findings from synthetic to naturalistic datasets, these catasetss illustrate the detrimental impact of increased statistical complexity SLM coherence.
Thus far, our dataset construction has been guided primarily by the desiderata outlined above. In the next section, we provide detailed quantitative evidence confirming that these datasets align with those goals. Section then presents the core experimental results.
3 Measuring and Validating Dataset Properties
To rigorously assess the properties targeted during dataset construction (Section 2), precise operationalization and quantitative validation are necessary. This section defines measur- able metrics for readability, statistical simplicity, and quality, empirically validates their suitability, and confirms that our datasets exhibit the intended characteristics.
3.1 Readability
Readability- ease with which humans comprehend text-is intuitive yet challenging to quantify automatically. While human judgment is the gold standard, its large-scale application is infeasible. We therefore evaluated three automated approaches: (1) classic readability formulas (e.g., Flesch- relying on surface features like sentence length and syllable counts; (2) constituency parsing metrics, assessing syntactic complexity; and (3) LLM-as-a-judge, using instruction-tuned LLMs to directly score readability (Figure A10).
To select the most appropriate metric, we correlated these automatic measures with human readability judgments from the CLEAR dataset (Heintz et al., 2022), as shown in Figure la. Large-scale LLMs 2aama-3.-70B-Imstrruc, Qwen2-72B-Instruce demonstrated the strongest correlation with human judgments (Pearson 0.74), substantially outperforming classic formulas (e.g., Flesch-Kined Kincaid Grade: r = 0.49) and parsing metrics (e.g., max tree depth: r = 0.34). Given its superior alignment with human perception, we adopt LLM-as-a-judge using Llama-3.1700BInnstru for readability measurement throughout our experiments. While acknowledging that LLM-based scoring not a perfect substitute for human evaluation and may carry its own biases, its strong empirical grounding on this benchmark makes the most suitable metric for our large-scale analysis.
3.2 Quality
Evaluating the quality of generated text is necessary for comparing models, but manual assessment is impractical due to subjectivity and scale. We therefore require automatic proxies. We explored two: perplexity, measuring text likelihood averaged across moderately sized models (Mistral-7B-v0.3, Qwen2-7B, Figure 1b), though lower perplexity doesn't guarantee higher overall text quality; and
-ufffiiiint per validation, see
3

---

Published as a conference paper at COLM 2025
Fastaaiiyy
Matru-33
2.0
Clossio
Pouc Cocud Tlaak
0:22
claec
2400
3.7
0.29
0:00
0.90
2:43
Construency Panso
3.56
0:67
0.99
344
0:09
0.57
0.89
2.79
0:59
6.11
8.64
11
6036 0.11
LLM as- Audge
3.33
8:48
LiMaseo-llaage
0:51
9.43
0.26 9.44
Masiud botrew
3:18
0.31
0.55
a.6
0.51 0.46 3.33
0.53
0.21
Jamma enstruct
8:1
0:43
0:31
0.24
0.37
3.8
839
0.34 0.49 0:00
0:23
0..7
tramnnal hitor 4a8
3:15
6.22
0.00
0.3
623
6.66
0.2
3:5
02
0.5
62 5166 0.77
Des O footbey
PARTO NOD
P 1006099
(a) Readability
(b) Quality
Figure 1: Validation of readability and quality metrics. (a) LLM-based readability scores correlate best with human judgments (r 0.74 via Llama-3. 1-l-1-------nnnn on CLEAR dataset), outperforming classic formulas and parsing metrics. (b) LLM-based coherence scores correlate best with our model ranking (Table A4), outperforming perplexity.
LLM-as-a-judge. For the latter, rather than using an ambiguous generic "quality" score, prompted an LLM (Figure A11) to assess coherence-logie structure and flow.
To select the more effective proxy, correlated both measures against a constructed ranking reflecting broad, recognized tiers of model generation capability (e.g., Llama 3.1 vs. GPT2; Table A4). This pragmatic reference helps validate which proxy better captures obvious quality differences. LLM-judged coherence demonstrated significantly stronger correlation with this ranking (Figure 1b) and minimal correlation with readability (Figure A27). While several large instruction-tuned models performed well coherence judges, we selected for consistency with the model used judge readability. Based on its strong empirical performance relative to perplexity and its orthogonality to readability, we adopt LLM-judged coherence as our primary quality metric.
Notably, while coherence was the specific dimension prompted, our core conclusions remain consistent when evaluating other relevant quality aspects. Supplementary analyses using the same LLM-as-a-judge method to assess dimensions like fluency and clarity yielded the same patterns regarding dataset effects.
3.3 Statistical Simplicity
We quantify statistical simplicity using n-gram diversity, computed over 1-gram to 8-gram sequences within each dataset. This metric captures how often certain token patterns repeat-a proxy for the dataset's distributional regularity, that is, the extent to which sequences are structured, repetitive, and predictable.
N-gram statistics provide a simple and interpretable way to characterize how structured or compressible dataset is. Because they reflect how often specific token sequences recur, they offer a useful lens into the dataset's underlying predictability-eseeee models that have limited capacity to track long-range dependencies.
for small
3.4 Validation of Dataset Construction
In Section 2, we described how our datasets were constructed using intuitive, common-sense decisions about prompt structure, vocabulary, and narrative framing-without wwithut relying on formal methodology or prior empirical validation. In this section, we evaluate whether
4

---

Published as a conference paper at COLM 2025
1e8
1.0
TinyStories
UamaTales--p
Uaaa DamaTales-GRE
Uama SplrssS
UamaTales-HHe History
UamaTales News
0.6
FineWeb
Unique
Slimpajaaa
0.2
0.0
1-grams
2-grams
3-grams
4-grams
5-grams
6-grams
7-grams
8-grams
Figure 2: Unique n-gram counts across datasets (100M token samples). The synthetic datasets (TinyStories, Lalea series) statistically simpler than standard pretraining corpora (FineWeb, Dolma, SlimPajama), as demonstrated by their substantially fewer unique n-grams (indicating lower diversity and higher predictability).
those intuitively constructed datasets actually exhibit the three properties we aimed to target: controlled readability, statistical simplicity, and consistent quality.
Statistical Simplicity Figure reveals two distinct clusters n-gram diversity, measured as the number of unique n-grams in 100M-token samples. Synthetic datasets (TinyStories and the LlamaTale series) exhibit markedly lower diversity than standard pretraining corpora such as FineWeb, Dolma, and SlimPajama. This separation confirms that our data generation pipeline successfully produced datasets with the intended statistical simplicity.
Table 1: Quantitative comparison of dataset properties. Metrics confirm controlled read- ability differences and high coherence within synthetic datasets. Top: Classic readability formulas. Mid: Constituency parsing. Bot: LLM-as-a-judge.
TinyStories
LlamaTales-Ia
LlamaTales-GRR
FineWeb
Automated Readability
2.9
2.9
12.4
13.1
Coleman-Liau
3.7
3.8
10.4
11.8
Dale-Chall
5.7
5.7
9.1
9.3
2.4
2.2
9.6
10.7
Gunning Fog
4.6
3.8
11.7
12.1
Linsear Write
4.2
3.3
13.2
12.7
SMOG
5.7
5.4
11.3
12.6
Spache Readability
2.7
2.5
5.5
5.5
Depth / Sentence
6.8
6.4
10.6
9.5
Width Sentence
5.1
4.7
8.0
7.5
Nodes / Sentence
19.6
17.2
42.1
37.8
Readability
92.6
92.7
64.8
68.2
Coherence
90.1
89.5
94.4
77.4
Controlled Readability Table 1 shows that our datasets vary systematically in readability. LlamaTales-GRRE constructed using GRE-level vocabulary, scores lower than LlamaTales- Jr and TinyStories across classic readability formulas, parsing metrics, and LLM-based judgments. This validates our ability to control for readability during dataset construction.
Consistent Quality Despite these differences, coherence remains consistently high across all synthetic datasets. FineWeb- deeb-despite having similar readability to LlamaTales- notably lower coherence. This likely reflects the noisier, less structured nature

---

Published as a conference paper at COLM 2025
of web-sourced datasets, even after extensive filtering by their original authors. In contrast, synthetic data generated using instruction-tuned models tends to be more consistently well-formed, reflecting their optimization for human-preferred outputs.
Having validated the properties of our datasets, we next examine how differences in readability and statistical simplicity affect model behavior. We train SLMs from scratch each dataset and evaluate their ability to generate coherent stories using prompts drawn from both in-distribution and out-of-dist test sets. This setup allows to assess whether and how these data properties influence coherence and generalization in small-scale models.
4 Results
Experimental Setup We train transformer language models from scratch on each dataset described in Section These models range in size from 262K to 33M non-embedding parameters. Each model is trained for 10 billion tokens across 10 epochs. We also evaluate several public pretrained models (e.g,, GPT-2, Pythia, Mistral, Qwen2, Llama-3) as baselines without fine-tuning them on our data (details in Table A2).
To assess model performance, we generate 1,000 completions per model-dataset pair using top-p sampling 0.95.. Prompts consist of 50-token excerpts drawn from the test split of each respective dataset. We evaluated these generations using several automated metrics. Following our validation in Section 3.4, LLM-judged coherence serves as our primary measure of generation quality for the core results presented in this section. Supplementary analyses using other metrics, including perplexity and additional LLM-judged dimensions (such as readability, fluency, clarity, consistency, and grammar), are shown in Figures A3-A9).
High Coherence Does Not Require Readable Text While datasets like TinyStories are often highlighted for their simplicity and readability, our results indicate that readability itself is not the necessary factor for coherence to emerge in SLMs. As shown in Figure 3, GREaaalll--GE--cieeee high in-distribution coherence scores. For instance, a 33M parameter model trained on LlamaTales-GRE (blue dots, Figure 3b) reaches a coherence score comparable that of Llama-3.1-70B when evaluated on
SLMs trained on synthetic data with low readability-speeiffaaalll
prompts. This level of in-distribution performance is on par with that achieved by models trained on high-readability Llama Tales- Jr (green dots, Figure 3a). This holds despite MammaTales-Cii text being significantly less readable. This pattern also persists across different narrative domains (news, sports, history), as shown in Figure A9, reinforcing the finding that human readability is not the primary driver for SLMs generating coherent text.
Readability Does Not Speed Up Coherence Emergence A potential refinement of the readability hypothesis is that simpler language might accelerate learning, mirroring devel- opmental stages where simpler input is often assumed beneficial. Our findings, however, suggest different dynamic during training. We tracked coherence development throughout training (Figure 4b). Models trained on the less readable, more complex Llama' GRE dataset (blue) achieve high level of coherence remarkably quickly, surpassing a score of 85 after just the first epoch, followed by more gradual improvement. In contrast, models trained on the simpler TinyStories (orange) or Llama coherence levels and exhibit much steeper learning curve over subsequent epochs, indi- cating they require more training data exposure to reach high coherence compared to the rapid initial learning facilitated by the complex LlamaTales- of high coherence when training complex text directly challenges the developmental framing; if simpler, child-directed language were inherently easier to learn from, we would expect it to enable faster, not slower, initial emergence of coherence.
(green) datasets start at lower
data. This rapid attainment
Linking Learnability to Statistical Simplicity If readability explains neither final coher- ence nor learning speed, what property does? We propose statistical simplicity, measured himpiiity, here via n-gram diversity, is the key factor. To investigate this, we define a learnability ratio: the coherence of model's output divided by the coherence of the dataset it was trained on.

---

Published as a conference paper at COLM 2025
100
100
1ome--.1-700
pytha 12b
U-1--111-70
Conerence
Conerence
pytha 12b
Train Data
hubli Model
20
20
dema
Thyyfrones Rnowee
gp12
Uomo 25 Jr
107
10
10
20th
20
2011
207
10
10th
20"
10
2011
Parametors
Embcdding Parametors
(a) 1lamaTales-JD prompt completions.
(b) L LlamaTales-GRE prompt completions.
Figure 3: LLM-judged coherence scores versus model size. Colors indicate training data; black indicates public reference models. High in-distribution coherence achievable regardless of readability, alongside strong distribution dependence. Red line: training data coherence for the respective prompt set. See Figure A2 for results for FineWeb and TinyStories.
Lammataaas one
Liama Siles Sports
Lome tas Hitoyy
Conerence
80
LiamaTales GRE
Usmataless-j
Tiyyttories
10
Un Cun 2111
167
spoch
(a) Learnability vs Unique 3-grams
(b) Coherence across training
Figure Statistical simplicity predicts learnability; readability does not determine learning speed. (a) Learnability ratio (output coherence / train data coherence) vs unique 3-grams, showing higher learnability for statistically simpler datasets (top-left). (b) Coherence dur- ing training. Note the rapid initial coherence emergence when training on less readable JlamaTalee-Cii (blue) compared high-readability datasets (green/ /orange).
A ratio near 1.0 indicates the model effectively captures the coherence present in its train data.
Figure 4a plots this learnability ratio against 3-gram diversity for models trained on various datasets. It reveals clear inverse correlation and two distinct regimes: SLMs trained on synthetic datasets (TinyStories, LlamaTales), which have low n-gram diversity, exhibit learnability ratios close to 1.0 (top-left). In contrast, models trained on standard pretraining data, which have much higher n-gram diversity, exhibit significantly lower learnability ratios (bottom-right), struggling to capture the coherence potential of their data. This suggests statistical simplicity influences not only the speed at which coherence is learned but also the extent to which models capture properties present in train data. Notably, this pattern is not explained by readability; LlamaTales-Cr and FineWeb have similar readability scores, yet models trained on Llama La Tales-GRE achieve much higher learnability due to its lower
7

---

Published as a conference paper at COLM 2025
statistical complexity. Taken together, these results strongly suggest statistical simplicity, not developmental readability, is the more reliable predictor of learnability in SLMs.
SLMs Trained on Synthetic Data Are Not Robust While SLMs trained on our synthetic datasets achieve impressive in-distribution coherence, their capabilities are narrow and highly distribution-specifff When evaluated on held-out prompts from the same dataset they were trained on (e.g, LlamaTales- GRE models on LlamaTales- prompts), these small models perform comparably to much larger models. However, when tested on prompts from any other dataset-includiig text-their coherence degrades substantially (e.g,, LlamaTales-GRR models perform poorly on LlamaTales- prompts). This brittleness likely stems from the very statistical simplicity that enables high in-distribution coherence; the models learn patterns specific to the narrow training distribution and fail to generalize beyond it. The primary exception involves TinyStories and LlamaTales-Jr which share near-identical generation processes and target audiences, resulting in some cross-dataset coherence. We did not explore training on mixed or broader synthetic distributions, which might mitigate this. Nonetheless, these findings underscore cautionary point: high coherence in SLMs trained on narrow synthetic data can give a misleading impression of general capability. As enthusiasm grows for small models, is crucial to distinguish in-distribution fluency from robust, generalizable understanding.
other synthetic corpora or real-world web
SLMs Are Not Merely Memorizing Training Data While the previous section highlighted the failure of these SLMs to generalize robustly outside their narrow training distribution, the question remains whether their high in-distribution coherence stems from genuine pattern learning or simply memorizing training sequences. Given the statistical simplicity of the datasets, this latter concern is particularly relevant. To assess this, we compute n-gram novelty-the proportion of n-grams in model outputs that do not appear in the training set- sel-following Merrill et al. (2024). As shown in Figure 5, SLMs trained on Llama Tales datasets generate substantial number of novel n-grams, particularly at mid-range lengths (e.g., 3- to 5-grams). This indicates their outputs are not simply copied but are generated by recombining learned distributional patterns. These results suggest that our models are indeed learning and recombining structure, albeit within the narrow confines of their training data, rather than merely memorizing.
Fyticoo 33M sycciieess
Brrewdr F3N
Berook- PM
Bonaein 1o
Uire 704
04
2.5
25
6.4
62-
(a) TinyStories
(b) LlamaTales-Jr
(c)
Figure 5: N-gram novelty (percentage of generated n-grams absent in the training data) vs. n-gram size for SLMs trained on synthetic datasets. Substantial novelty indicates recombination beyond memorization. Llama-3.1-70B (black dots), serving as baseline, was not trained on these datasets; its generations are compared against each training set here.
Synthesizing these results, we find that SLMs can learn coherent structure effectively from statistically simple data, irrespective of readability, but this learned capability remains narrow and fails to generalize robustly beyond the specific training distribution.
5 Discussion and Conclusion
Our results offer key clarification regarding the emergence of coherence in small language models: human readability, often emphasized in discussions surrounding datasets like TinyStories and framed developmentally, is not the decisive factor. We find that SLMS
8

---

Published as a conference paper at COLM 2025
trained complex, less readable text can achieve comparable or even superior coherence, sometimes learning it more rapidly than models trained on simpler, child-directed corpora. This directly challenges the intuition that human-centric linguistic simplicity inherently facilitates learning in these systems.
What, then, enables coherence? Our findings point strongly towards statistical simplic- thy-the predictability and structural regularity of the training data distribution as the more reliable predictor of learnability. While we operationalize this using n-gram diversity as a proxy, this metric likely captures one facet of a broader set of properties (e.g., com- pressibility, structural consistency, limited long-range dependencies) that make the data statistically easier for small models to learn effectively. It is this underlying predictabil- ity, rather than surface-level readability for humans, that appears to play a central role in efficient coherence acquisition in SLMs.
Anthropomorphic metaphors such as models 'learning like children' or synthetic datasets 'mimicking developmental input how researchers frame small model training, particularly in developmentally inspired data design. As Ibrahim & Cheng (2025) highlight, such framings influence research questions and methodologies. Often, this involves im- plicitly or explicitly treating characteristics associated with high human readability (e.g., accessible vocabulary, simple syntax) as the key driver of success. However, this risks conflating this human-centric simplicity with statistical learnability (the ease of modeling the data's ss distribution), obscuring the actual mechanisms driving capability emergence and potentially misdirecting research efforts.
This metaphorical framing reflects broader tendency to attribute cognitive qualities to systems that lack them. As Shanahan (2023) emphasizes, language models remain statistical predictors-traieee trained generate likely continuations of text-not agents with access to meaning, truth conditions, or communicative intent. Using terms like "belief" or "learning" to describe their behavior risks conflating linguistic fluency with cognitive competence, inviting interpretations these systems do not warrant.
These issues are not limited to research practice. As Placani (2024) argues, anthropomorphic framing also shapes how language models are perceived in public discourse and policy. can lead users to overestimate model's understanding while deflecting responsibility from those who design, train, and deploy them. When models are described as learning or developing, they may appear more autonomous than they are, shifting blame for errors from the pipeline to the system. These misconceptions complicate efforts to ensure transparency, accountability, and effective governance.
Interest in developmentally inspired language model training is growing, exemplified by efforts like the BabyLM Challenge (Warstadt et al., 2023), which promotes cognitively scientifically valuable,
plausible data, constraints, and interactions. These initiatives especially when framed as intentional biomimicry or used to generate testable hypotheses. However, our findings offer a cautionary counterpoint: coherence in small models does not require child-directed inputs or simplified language, as might be assumed under purely developmental framing.
emerges from training on synthetic corpora that are statistically simple even when they are less readable. In this context, developmental resemblance should not be mistaken for causal explanation. We encourage future work, in line with critiques of anthropomorphie influence on methodology (Ibrahim & Cheng, 2025), to carefully distinguish between potentially useful conceptual framings (like developmental analogies) and the empirically validated mechanisms that actually support learnability.
Some may see our findings as reaffirming something already understood that language models are statistical learners, and that low-complexity data is easier to learn from. But the continued presence of readability-orieme inspired datasets like TinyStories, suggests that this clarification remains timely. As Ibrahim & Cheng (2025) note, anthropomorphic language is increasingly common, influencing both research narratives and methodological choices. In this context, reaffirming statistical fundamentals helps restore clarity. We do not reject metaphor analogies can guide intuition they must be distinguished from empirical explanations.
framing, especially in work on developmentally

---

Published as a conference paper at COLM 2025
Framing model behavior in developmental terms risks obscuring the actual factors that support learning.
This statistically grounded perspective points towards productive future research. Key questions include developing richer, more comprehensive measures dataset complexity and learnability beyond simple n-grams, designing training curricula data selection strategies that leverage statistical simplicity without sacrificing generalization, and creating evaluation methods that effectively disentangle genuine capabilities from pattern matching facilitated by statistically simple data structures.
In conclusion, we propose reframing: the emergence of coherence in small models trained on specific datasets is not evidence of achieving a human-like developmental milestone. Rather, is primarily statistical outcome reflecting the model's success learning patterns from distributionally simple and coherent data. Recognizing this distinction is important for accurately understanding SLM capabilities and for aligning model design, research framing, and public interpretation with the mechanisms that actually govern their behavior.
References
Bastian Bunzeck and Sina Zarrieß. GPT-wee: How small can a small language model really get? In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyL.M Challenge at the 27th Conference on Computational Natural Language Learning, pp. 35-46, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10.18653/v /22/5///////2.....//// cob..oonllbaa URL https: / https://aclanthologg org/2022 conll conl1-bbylm..2 2/.
Meri Coleman and Ta Lin Liau. A computer readability formula designed for machine scor- ing. Journal of Applied Psychology, 60:283-284, 1975. URL https: //api. semanticscholar. org/CorpusID: 44250124.
Edgar Dale and Jeanne Sternlicht Chall. A formula for predicting readability. 1948. URL https://api. semanticscholar wrwpaaarorr/C/ooooDD.0 19AD::1908000 190836965.
Lukas Edman and Lisa Bylinina. Too much information: Keeping training simple
for babylms. ArXiv, abs/2311.01955, 2023. URL http:://app..emmtticcchhete www.ntccscooolrr........ .org/
CorpusID:2659885194
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.orge wttp:/wwwwwwrr////////////////// 07759.
Steven Y. Feng, Noah D. Goodman, and Michael C. Frank. Is child-directed speech effective training data for language models? ArXiv, abs/2440.006177, 2024. URL https://api. semanticscholar.. wwwphhollrrrrr//////e/ ant//CorpuID::21744400
Robbie Gunning. The technique of clear writing. 1968. URL htt.t::://ap....mmticceceeeeeeee :1CtorpusID:145888880
Akari Haga, Saku Sugawara, Akiyo Fukatsu, Miyu Oba, Hiroki Ouchi, Taro Watanabe, and Yohei Oseki. Modeling overregularization in children with small language models. In Annual Meeting of the Association for Computational Linguistics, 2024. URL htaps://api.
semanticscholar.c /otp://ooooo:/11111555
Harry and Me Laughlin. Smog grading new readability formula. The Journal of Reading, 1969. URL httpss//app httpp::/api..semattischhee wwwanticscoooarrrrmmmmo mtg//orppuID:997775..
Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mehrnoush Karimi, and Agnes Malatinszky. A large-scaled corpus for assessing text readability. Behavior Research Methods, 55, 03 2022. doi: 10.3758 10.3758/s13428-000 11777/8811112220020011112xx
Lujain Ibrahim and Myra Cheng. Thinking beyond the anthropomorphic paradigm Il research, 2025. AtR.sttss://axiiv org/abs abs///a/////..// 09192.
paradii benefits
10

---

Published as a conference paper at COLM 2025
Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975. URL https://api semanticscholar. org// CorpusID:611313222
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024. URL
co/ wwt.tattss//uugggggga//////eiine
William Merrill, Noah A. Smith, and Yanai Elazar. Evaluating n-gram novelty of language models using rusty-dawg, 2024. URL httpp:://aaaeee http://aaxii...rrra/////////// wwt.:/r////b/...... 13869.
Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, and Anna Rumshisky. Emergent abilities in reduced-scale generative language models. ArXio, abs/2404.02204, 2024. URL
O'Hayre. Gobbledygook Has Gotta Go. U.S. Department of the Interior, Bureau of Land
Management, 1975. URL htttp:://boll google. httig/l..combooos../ comm/bookssid=I90kkeaee
Adriana Placani. Anthropomorphissm in ai: hype and fallacy. AT Ethics, 4:691-698, 2024. 267505422.
URL /ttps// http://pp...mmmtttccccoooa...rrr
Murray Shanahan. Talking about large language models, 2023, URL httpt:://///xii.orrr
abs/2212.0355 035511.
A Smith and R. Senter. Automated readability index. AMRL-TR, Aerospace Medical Research Laboratories, pp. 1-14, 1967. URL https: httpppp.ssaattscccollaaaaaaa org :tte///oppsI::::::00 38558516.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.
SlimPajama: A 627B token cleaned RedPajama. URL /tt.lt/ttt////ggggggaeeeeeettt/e
and deduplicated version aammaajama-a-62b-ttooe
https://cerebrat ai/blog/ 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024. URL https://arxivv orgg 00159.
George Daniel Spache. A new readability formula for primary-grade reading materials. The Elementary School Journal, 53:410 - 413, 1953. URL hittps://apt semanticscholar.orgg
CorpusID: 1451 15468.
Nikitas Theodoropoulos, Giorgos Filandrianos, Vassilis Lyberatos, Maria Lymperaiou, and Giorgos Stamou. BERTtime stories: Investigating the role of synthetic story data in lan- guage pre-training. In Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.), The 2nd BabyLM Challenge at the 28th Conference on Computa- tional Natural Language Learning, Pp. 308-323, Miami, FL, USA, November 2024. Associa- tion for Computational Linguistics. URLhttps: htt.tp:://aclaanhoooooo hwt.e/24.g//022..... conll babbyl... 28/.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.). Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Singapore, December 2023. Association for Computational Linguistics. URL https://acclanthologg.. wwwp:yyy..ro./o..o.ee// con 11 babylm. 0.
11

---

Published as a conference paper at COLM 2025
Hong Meng Yam and Nathan Paek. What should baby models read? exploring sample- efficient data composition on model performance. In Michael Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.), The 2nd BabyLM Challenge the 28th Conference Computational Natural Language Learning, pp. 284-291, Miami, Fl., USA, November 2024, Association for Computational Linguistics. URL https: hptp:://aaatthllooyy org/20244 con1ecbbblb-bbbn 25/.
A Classic Readability Formulas
The readability formulas utilized in Table 1 are presented below. The simplest way to measure readability is through formulas, many of which have been developed over the years. These formulas are generally straightforward, focusing on various combinations of word, sentence, and syllable counts. We employ textstat? to calculate readability using various established formulas, including FKGL.
A.1 Flesch-Kincaid Grade Level (Kincaid et al., 1975)
words sentences
syllables 15.59 words
FKGL = 0.39
+ 11.8
(1)
A.2 Automated Readability Index (Smith & Senter, 1967)
[characters
words
ARI 4.71
+ 0.5
-21.33
(2)
words
sentences
A.3 Coleman-Liau Index (Coleman & Liau, 1975)
characters x 100) - 0.296
sentences x 100) 15.8 words
CLI = 0.0588
(3)
words
A.4 Dale-Chall Formula (Dale & Chall, 1948)
(difficult words words
words sentences
DC = 0.1579
x 100 + 0.0496
(4)
Difficult words are defined those not included in list of 3,000 words that fourth-grack American students are expected to know. If the percentage of difficult words exceeds 5%, add 3.6365 to the score.
A.5 Gunning Fog Index (Gunning, 1968)
words
/complex words }
GFI = 0.4
+ 100
(5)
sentences
words
Complex words are defined as words with three or more syllables, excluding proper nouns, familiar jargon, and compound words.
A.6 Linsear Write Formula (O'Hayre, 1975)
(words < syllables) + (words > 3 syllables)
(6)
sentences
if > 20
Linsear Write
(7)
if 5 20
12

---

Published as a conference paper at COLM 2025
A.7 SMOG Index (Harry & Laughlin, 1969)
SMOG 1.0430 30 words 2 3 syllables + 3.1291
(8)
sentences
A.8
Spache Formula (Spache, 1953)
words + 0.082
/difficult words
Spache 0.121
100 + 0.659
(9)
sentences
words
Difficult words are defined as words that are not included in list of familiar words that are typically known by -ourth-grade students.
S
NP
VP
DT The
NN
VBD
PP
cat
sat
IN
NP
on
DT
NN
the
mat
Figure A1: Constituency parse tree for the sentence
"The cat sat on the mat."
B
Additional Figures and Tables
Dataset
Coherence
Readability
TinyStories
90.1
92.6
LlamaTales-Jr
89.5
92.7
LLamaTales-GRE
94.4
72.7
LlamaTales--ports
92.4
72.4
LlamaTales-Newt
94.5
72.7
Tamma
91.0
61.4
FineWeb
77.6
68.2
Dolma
60.2
70.99
Sl imPajaaa
67.3
66.6
Table A1: Coherence and readability scores for train splits.
13

---

Published as a conference paper at COLM 2025
60
Wam0-111-700
Lama-1111 1.1. 706
Conerence
pythra122b
40
PytNa-120
40
20
gpi2-al
gpt2-1
20
10
20
201
10
10
20
10
Non Embedding Parameters
Non Eobbemdeddingg Parameters
(a) TinyStories prompt completions.
(b) LlamaTales-Da prompt completions.
80-
Lame-331-70B
sythie-12b
60
60
Contrence
Conerence
Llama- 1.1- 708
pyb
40
gp12-al
gptcis
Putali
Jama aa ORE
"inySronee
302
Nor-Embedding Parameters
Ron-Embedding Parameters
(c) LaamaTales-GGE prompt completions.
(d) FineWeb prompt completions.
Figure A2: Coherence of text generated by the LMs listed in Table A2. Prompts extracted from the test splits of our datasets in Table 1. The legend colors represent the training data for each model. Public models (black) are found on Huggingface and are not fine-tuned on our data. The red horizontal line marks the coherence of the train split of the dataset in focus. Return to Figure (truncated results).
14

---

Published as a conference paper at COLM 2025
10T
107
Perpiextt
Perplexity
10²
pythis-126
10'
pythia
3am
327
187
107
104
30?
1021
10
107
1630
1022
Mon-Embeddaing Panameters
Mon-Embeddding Panameters
(a) TinyStories prompt completions.
(b) LlamaTales-Jr prompt completions.
Usmrattles GRE
10'
10'
They Strrien
Peplixit
apt2-xl
pyth
pythia 12b
162
1nl
Llama
Lama Lamm--311700 3.1-70B
300
107
107
20
202
2030
20
10a
207
10t0
Non Embedding Parameters
Non Embedding Parameters
(c) LlamaTales-GRE prompt completions.
(d) FineWeb prompt completions.
Figure A3: Perplexity (computed by external LMs) of text generated by the LMs listed in Table A2, based on prompts from the test split data in Table 1. See Section 3.2 for details on this metric. The legend colors represent the training data for each model. The red horizontal line marks the perplexity of the train split of the dataset in focus. Return to Section (Results).
15

---

Published as a conference paper at COLM 2025
Ul-00-111700
Llama-3.1 3-1708
pythia 12b
Pythi=122
60
80
Reatsabiityy
Roaiiabiityy
gpt2zd
get2 xi
20
20
10
30at
20ah
10a
107
1010
Non-Embeddding Parameters
Non-Embedding Parameters
(a) TinyStories prompt completions.
(b) LlamaTales-Jr prompt completions.
100
100
80
80
Ulema-3 1-708
Reaisbilyy
pythie-120
Roadsabliyy
pythia-12h
gpt2
gpt2xxl
40
Pubric
Storlc
Unn lonalales-on
10"
207
10at
20*
10
20
10a
107
20*
10ta
10"
Non-Umbeddiing
Non-Umbelding
(c) LlamaTales-GRE prompt completions.
(d) FineWeb prompt completions.
Figure A4: Readability of text generated by LMs in Table A2, based on prompts from the data (test split) in Table 1. The legend colors represent the training data for each model. The red horizontal line marks the readability of the train split of the dataset in focus. Return to Section (Results).
16

---

Published as a conference paper at COLM 2025
80
80
Hamm-1 1-70R
Pythis-12b
Fluerry
pythie
GpF2
gpt2
20
20
10la
10
10l6
Non-Em eon-Embeddding Panameters
Non-Em Kon-Embedding Parameters
(a) TinyStories prompt completions.
(b) LlamaTales-Jr prompt completions.
100
100
Jama-3.1 1.1-70R
Pythia 12b
60
gpt2 xi
60
pythia
Plaorc
liyySturiee
10
307
30a
20a
10
10
10a
107
10
20a
1016
10"
Non-Embeddding Panameters
Non--mbeedding Petameters
(c) LlamaTales-GRE prompt completions.
FineWeb prompt completions.
Figure A5: Fluency of text generated by LMs Table A2, based on prompts from the data (test split) in Table 1. The legend colors represent the training data for each model. The red horizontal line marks the fluency of the train split of the dataset in focus.
17

---

Published as a conference paper at COLM 2025
80
80
Llama-3.1-708
pythia
pythio
Caily
20
20
gpt2-al
10la
10
10l6
Non-Emiin egn-Embeddding Parameters
Kon-Embedding Parameters
(a) TinyStories prompt completions.
(b) LlamaTales-Jr prompt completions.
100
100
Lam--311-700
Laaa-3311700
60
pythis-12b
60
Caily
pythia 12b
y142
10
307
30a
20a
10
10
10a
107
10
20a
1016
10"
Non-Embeddiing Panameters
Non--mbeedding Petameters
(c) LlamaTales-GRE prompt completions.
(d) FineWeb prompt completions.
Figure A6: Clarity of text generated by LMs in Table A2, based on prompts from the data (test split) in Table 1. The legend colors represent the training data for each model. The red horizontal line marks the clarity of the train split of the dataset focus.
18

---

Published as a conference paper at COLM 2025
Ummi-3117788
80
80
Comsisency
Commisteny
pythie1120
20
20
10la
10
10le
Non-Emiin eon-Embeddding Parameters
Non-En Non-Embeddingg Parameters
(a) TinyStories prompt completions.
(b) LaamTTales-J prompt completions.
100
100
Llama-311-700
60
60
concimencyy
consistrncy
pythie-12b
grf wi
liyySturise
10
30
30at
20a
10
20
10"
107
10f
20a
1016
10"
Non-Embeddiing Panameters
Non-tmbeedding Petameters
(c) LlamaTales-GRE prompt completions.
(d) FineWeb prompt completions.
Figure A7: Consistency of text generated by LMs in Table A2, based on prompts from the data (test split) in Table 1. The legend colors represent the training data for each model. The red horizontal line marks the consistency of the train split of the dataset focus.
19

---

Published as a conference paper at COLM 2025
100
100
708
prthia-1bb
gybsa-12b
gp12 al
orannerr
gp42
Grammnrr
40
40
20
Non-Embelding Parameters
Non-Embedding Parameters
(a) TinyStories prompt completions.
(b) LamaTales-Jr prompt completions.
100
100
pythia-12b
Unmee
1-700
got2zel
pythia-12b
Grammar
Grammar
40-
RncWee
UnsmaThilee cas
Uamoliales
109
107
20
10f0
2011
20*
102
1010
2014
Embedaing Paremeters
Non Embeddind
(c) LlamaTales-GRE prompt completions.
(d) FineWeb prompt completions.
Figure A8: Grammar of text generated by LMs in Table A2, based on prompts from the data (test split) Table 1. The legend colors represent the training data for each model. The red horizontal line marks the grammaticality of the train split of the dataset in focus.
20

---

Published as a conference paper at COLM 2025
80
Jem033.1-700
Llan--1. 1-100- 1-708
Colference
Colterence
psthis-12b
Sptz-al
40
Jamatties New
Stmppapma
simpajjma
Jamta ldie-tittry
Dolia
Dumma
lbonalbiee sichary
TomesTiee Sponss
107
100
20
1220
10
107
10f
10a
10
20
Ron-Embedding
Ron-Fmbbddinng
(a) Lamaaalls-SSprts prompt completions.
(b) LlamaTales-News prompt completions.
1lama-3.1-700
Cohference
60
pythi-12b
Coherence
pythis-12b
Thain Dats
Train Data
Puslic Model
SpiZ-al
Public Model
Llanalakezeiion
Dolma
20
20
Simpojama
Usisstics
Embedding Parameters
Non Embedding Parameters
(c) LlamaTaallaaaaaaa alemaTTaleittstory prompt completions.
(d) Dolma prompt completions.
100
Coherence
pythib
40
Data
d get2-al
Mippaammo
Ubmsiilles News
UsmssRiee History
107
1oa
169
1030
1641
Embodding Parametars
(e) SlimPajama prompt completions.
Figure A9: Coherence of text generated by LMs in Table A2, based on prompts from the data (test split) in Table 1. The legend colors represent the training data for each model. The red horizontal line marks the coherence of the train split the dataset in focus.
21

---

Published as a conference paper at COLM 2025
System Prompt:
You are an experienced teacher, skilled at identifying the readability of different texts.
User: Read the text below. Then, indicate the readability of the text, on scale from 1 (extremely challenging to understand) to 100 (very easy to read and understand). In your assessment, consider factors such as sentence structure, vocabulary complexity, and overall clarity.
<Text></Text>
On scale from (extremely challenging to understand) 100 (very easy to read and understand), how readable is this text?. Please answer with single number.
Figure A10: Prompt used for LLM-as-a-Judge to evaluate readability. See Section 3.1.
System Prompt:
You are experienced teacher, skilled at identifying the coherence of different texts.
User: Read the text below. Then, indicate the coherence of the text, scale from 1 (extremely incoherent) to 100 (very coherent). Remember that coherent text should be well-structured and well-organized. Coherent text should not just be heap of related information, but should build from sentence to sentence.
<Text></Text>
On a scale from 1 (extremely incoherent) to 100 (very coherent), how coherent is this text?. Please answer with single number.
Figure A11: Prompt used for LLM-as-a-Judge to evaluate coherence. See Section 3.2
System Prompt:
You are an experienced teacher, skilled at identifying the coherence of different texts.
User: Read the text below and evaluate the coherence of the text. Remember that coherent text should be well-structured and well-organized. Coherent text should not just be a heap of related information, but should build from sentence to sentence. <Text></Text>
Please provide a short analysis of the text's coherence. After your analysis, on a scale from (extremely incoherent) 100 (very coherent), how coherent this text? Please answer with single number.
Figure A12: Prompt used for LLM-as-a-Judeee to evaluate coherence, instructing the LM to generate an analysis before generating a score.
22

---

Published as a conference paper at COLM 2025
System Prompt: You experienced teacher, skilled at identifying the coherence of different texts.
User: First, consider the following examples:
Positive Example (Very Coherent):
The process photosynthesis is essential for plant life. begins when sunlight is absorbed by chlorophyll in the leaves. This energy is then used to convert carbon dioxide and water into glucose and oxygen. The glucose provides energy for the plant, while the oxygen is released into the atmosphere.
This text is coherent because it is well-structured, with each sentence building on the previous one to explain a process clearly and logically. Negative Example (Incoherent):
Photosynthesis is process. Leaves are green. Oxygen is in the air. Plants need water. Sunlight is bright.
This text is incoherent because it lacks logical flow and structure, presenting disjointed facts without clear connections progression.
Now, read the text below and evaluate its coherence on a scale from (extremely incoherent) to 100 (very coherent). Remember that coherent text should be well- structured and well-organized, not just a heap of related information. <Text></Text>
Please provide a short analysis the text's coherence. After your analysis, on a scale from (extremely incoherent) to 100 (very coherent), how coherent is this text? Please answer with single number.
Figure A13: Prompt used for LLM-as-a-Judge to evaluate coherence. This version includes positive and negative examples for reference.
System Prompt:
You are an experienced teacher, skilled at identifying the readability
of different texts.
User: Read the text below and evaluate the readability of the text. In your assessment, consider factors such sentence structure, vocabulary complexity, and overall clarity. <Text></Text>
Please provide short analysis of the text's readability. After your analysis, on scale from 1 (extremely challenging to understand) to 100 (very easy to read and understand), how readable is this text? Please answer with single number.
Figure A14 Prompt used for L.LM-as-a-Judge to evaluate readability, instructing the LM to generate an analysis before generating score.
23

---

Published as a conference paper at COLM 2025
System Prompt:
You are an experienced teacher, skilled at identifying the readability
of different texts.
User: First, consider the following examples:
Positive Example (Very Readable):
The cat sat on the mat. It was sunny day, and the cat enjoyed the warmth. The mat was soft and comfortable, making the perfect spot for nap.
This text is easy to read because it uses simple sentence structures, familiar vocabulary, and conveys ideas clearly. Negative Example (Challenging to Read):
In the midst of the diurnal cycle, the feline quadruped positioned itself upon the textile floor covering, basking in the solar radiance, which permeated the atmosphere with thermal energy, rendering the environment conducive to somnolence.
This text is challenging to read due to complex sentence structures, advanced vocabulary, and convoluted expression of ideas.
Now, read the text below and evaluate its readability on a scale from (extremely challenging to understand) to 100 (very easy read and understand). In your assessment, consider factors such as sentence structure, vocabulary complexity, and overall clarity.
<Text></Text>
On scale from (extremely challenging to understand) 100 (very easy to read and understand), how readable is this text?. Please answer with single number.
Figure A15: Prompt used for LLM-as-a-Judge to evaluate readability. This version includes positive and negative examples for reference.
24

---

Published as a conference paper at COLM 2025
You are experienced teacher, skilled at identifying grammatical errors of different texts.
System Prompt:
User: Read the text below. Then, indicate the grammaticality of the text scale from (extremely ungrammatical) to 100 (perfectly grammatical). In your assessment, consider factors such spelling, part of speech, sentence structure, punctuation, and overall grammatical correctness. <Text></Text>
On scale from (extremely ungrammatical) to 100 (perfectly grammatical) how grammatical is this text?. Please answer with single number.
Figure A16: Prompt used for LLM-as-a-Judge to evaluate grammaticality.
You are an experienced linguist, skilled at evaluating the fluency of different texts.
System Prompt:
User: Read the text below. Then, indicate the fluency of the text, on scale from 1 (poor fluency) to 100 (excellent fluency). your assessment, consider factors such as grammatical correctness, naturalness of language, and overall smoothness.
<Text></Text>
On scale from (poor fluency) to 100 (excellent fluency), how fluent is this text?. Please answer with a single number.
Figure A17: Prompt used for LLM-as-a-Judge to evaluate fluency.
System Prompt:
You are experienced teacher, skilled at identifying the consistency
of different texts.
Read the text below. Then, evaluate how consistent the first two sentences are with the rest of the text, on a scale from (extremely inconsistent) to 100 (very consistent). Consistent text should maintain logical flow and alignment in terms of theme, tone, and information throughout. <Text></Text>
User:
On a scale from (extremely inconsistent) to 100 (very consistent), how consistent the first two sentences of this text with the rest of the text? Pleasse answer with single number.
Figure A18: Prompt used for LLM-as-a-Judge to evaluate consistency.
25

---

Published as a conference paper at COLM 2025
You are an experienced teacher, skilled at identifying the clarity of different texts.
System Prompt:
User: Read the text below. Then, indicate the clarity of the text, on a scale from 1 (not clear at all) to 100 (extremely clear). In your assessment, consider factors such as coherence, conciseness, and comprehensibility.
<Text></Text>
On scale from 1 (not clear at all) to 100 (extremely clear), how clear is this text? Please answer with a single number.
Figure A19: Prompt used for LLM-as-a-Judge to evaluate clarity.
26

---

Published as a conference paper at COLM 2025
System Prompt: You are celebrated children's s author. You write stories that both easy to read and grammatically correct.
User: Write short story (3-5 paragraphs) which only uses simple words that 5 year old child would understand. The story should use the words: <WORD-1>, <WORD-2>, and <WORD-3>, The story has the following features: <FEAT-1> <FEAT-K>
Figure A20: Prompt used to generate LlamaTales-Jr, See Section 2.
System Prompt: You are a renowned fiction writer, celebrated for your imaginative storytelling and compelling characters. Your work spans various genres, including fantasy, science fiction, and contemporary fiction, and is known for its vivid de- scriptions, intricate plots, and emotional depth. Your writing is best appreciated by readers with the vocabulary and comprehension expected of college graduate.
User: Write short story (3-5 paragraphs). The story should use the words: <WORD-1>, <WORD-2>, and <WORD-3>. The story has the following features: <FEAT-1> <FEAT-K>
Figure A21: Prompt used to generate LLamaTales-GRe See Section 2.
You are distinguished historian, celebrated for your meticulous research and engaging narratives. Your work spans various historical periods and known for its depth, accuracy, and insightful analysis. You have talent for bringing history to life, making complex events and figures accessible and compelling to a broad audience. Your writing is best appreciated by readers with a keen interest in history and a desire to understand the past in a nuanced and comprehensive manner. User: Write short historical article (3-5 paragraphs) that provides insightful analysis of a significant event or figure. Include key details, context, and the impact on subsequent history. The story should use the words: <WORD-1>, <WORD-2>, and <WORD-3>. 3>. The story has the following features: <FEAT-1>
System Prompt:
<FEAT-K>
Figure A22: Prompt used to generate LlamaTales
27

---

Published as a conference paper at COLM 2025
System Prompt: You are experienced sports journalist known for your vivid and engaging coverage of athletic events and athletes' stories. Your writing captures the excitement, drama, and human elements of sports, appealing to both die-hard fans and casual readers. You have a keen eye for detail, a deep understanding of various sports, and the ability to convey complex strategies and statistics in accessible manner. Your articles are characterized by their dynamic prose, insightful analysis, and ability to place sporting events within broader cultural and social contexts. User: Write a short sports article (3-5 paragraphs) about recent game, match, or athletic performance. Include vivid descriptions, key statistics, and quotes from players or coaches if applicable. The story should use the words: <WORD-1>, <WORD-2>, ->, and <WORD-3>. The story has the following features: <FEAT-1> <FEAT-K>
Figure A23: Prompt used to generate LlamaTales-Spont
You are seasoned journalist at The New York Times, known for your incisive reporting and compelling storytelling. Your work covers wide range of topics, from breaking news and investigative journalism to in-depth features and opinion pieces. You have keen eye for detail, commitment to accuracy, and the ability to convey complex issues in clear and engaging manner. Your writing is characterized by its clarity, depth, and ability inform and engage a diverse readership.
System Prompt:
User: Write concise news article (3-5 paragraphs) about recent significant event. Include key details, quotes from relevant sources, and the broader context of the event. The story should use the words: <WORD-1>, <WORD-2>, and <WORD-3>. The story has the following features: <FEAT-1>
<FEAT-K>
Figure A24: Prompt used to generate LlamaTa Mammaaales-Nee.
TinyStories
Tales-Je
100000
Uama Tales-GRE
UamaTales- Splrr-
UamaTales History
Unique Counts
UamaTales News
FineWeb
60000
slimpajama
40000
1-grams
Figure A25: Unique unigram counts for 100M-token samples from each dataset. Refer to Figure 2 for higher values of n.
28

---

Published as a conference paper at COLM 2025
TinyStories
UamaTales-jr
Llama Sales-Sports
UamaTales-Hisoon History
Unique Counts
UamaTales News
0.8
FineWeb
Dolma
0.6
Slimpajama
0.4
2-grams
Figure A26: Unique bigram counts for 100M-token samples from each dataset. Refer to Figure 2 for higher values of n.
29

---

Published as a conference paper at COLM 2025
Table A2: Overview of the models used our experiments. Example generations from each model are shown in Tables A6 to A20. Top: Models trained from scratch on the dataset indicated by each model's prefix. Bottom: Pretrained models sourced from Huggingface. Return Section 4.
Model
Parameters
Train Data
Train Tokens
Layers
Heads
Model Dim
tinystories-2635
2.63e+05
TinyStories
le10
2
128
tinystories-524K
5.25e+05
TinyStories
1e10
2
128
tinystories-1M
1.05e+066
TinyStories
1e10
2
128
tinystories-9M
9.44e+06
TinyStories
1e10
4
6
384
tinystories-18M
1.89e+07
TinyStories
1e10
8
6
384
tinystories-33M
3.36e+07
TinyStories
1e10
8
8
512
Siniatalessjr-2660
2,63e+05
LiamaTales-Jr
1e10
2
128
llamatales jr-524K
5,25e+05
LlamaTales-Jr
1e10
2
2
128
Lamatales.jjr-M
1.05e+06
LlamaTales-Jr
1e10
4
2
128
1lmatalees_jr-9M
9.44e+06
Ll LlamaTales-Jr
1e10
4
6
384
1amatalees.jr-18M
1.89++07
Ll LamaTales-Jr
1e10
8
6
384
Lamatalees.jr-33M
3.36e+07
LlamaTales-Ih
1e10
8
8
512
11 lmmatales_gre-262X
2.63e+05
LamaTalles-Gr
1e10
2
128
11 amattales.gre-524k
5.25e+05
Llama lamaTales-GRE
1e10
2
2
128
11 Lmmatales.gre-1M
1.05e+06
LlamaTales-GRE
1e10
4
2
128
11 1amatales.gre-9M
9.44e+06
LlamaTales Tama-ale
1e10
6
384
11 amatallss_gre-1aM
1.89++07
LlamaTales--a
1e10
8
6
384
11 lamatales.gre-22
33M
3.36e+07
LlamaTales-Ga
1e10
8
8
512
fineweb-262K
2.63e+05
FineWeb
1e10
2
128
fineweb-524K
5.25e+05
FineWeb
1e10
2
2
128
fineweb-1M
1.05e+06
FineWeb
1e10
4
2
128
fineweb-9M
9.44++06
FineWeb
1e10
4
6
384
fineweb-18M
1.89e+07
FineWeb
1e10
6
384
fineweb-33M
3.36e+07
FineWeb
1e10
8
512
gpt2
8.51e+07
12
12
768
gpt22medium
3.02e+08
24
16
1024
gpt2-large
7,08e+08
36
20
1280
gpt2-x1
1.48e+09
48
25
1600
pythia-70m
1.89c+07
The Pile
3e11
6
8
512
pythia-168m
8.51e+07
The Pile
3e11
12
12
768
pythia-418m
3.02e+08
The Pile
3e11
24
16
1024
pythia-1b
8.06e+08
The Pile
3el1
16
8
2048
pythia-1.4b
1.21e+09
The Pile
3el1
24
16
2048
pythia-2.8b
2.52e+09
The Pile
3e11
32
32
2560
pythia-6.9b
6.44e+09
The Pile
3el1
32
32
4096
pythia-12b
1.13e+10
The Pile
3ell
36
40
5120
TinyL.ama.v1.
9.69e+08
SlimPajama
2e12
22
32
2048
Mistral-78-v0.3
6.98e+09
32
32
4096
Mixtral-8x78-vo
4.64e+10
32
32
4096
Qwen2-7B
6.53e+09
7e12
28
28
3584
Qwen2-728
7.02e+10
7e12
80
64
8192
Llama-3. 1.188 1-88
6.98e+09
15e12
32
32
4096
11am8-3.1-709 1-708
6.85e+10
15e12
80
64
8192
30

---

Published as a conference paper at COLM 2025
Table A3: Statistics for the train splits of the datasets described in Section 2 as well as (absolute) Pearson correlation coefficients against human judgments of readability (t is better). Random examples from each dataset are shown in Table A3. Top: Classic readability formulas (4 is easier to read). Mid-Top: Statistics computed from running constituency parser over the sentences of each dataset (f suggests more grammatical complexity). Mid- Bot: The result of prompting Llama-3. 1-708-Instruuc to judge readability and coherence (f is better). Perplexity is computed with Llama-3. 1-8B, Qwen2-7B, and Milstral-7B-ve and averaged. Bot: Word, token, and syllable level statistics. Truncated statistics are shown in Table 1.
Corr.
TinyStories
LlamaTales-Jr
LlamaTales-Gre
FineWeb
Automated Readability
0.47
2.9
2.9
12.4
13.1
Coleman-Lian
0.48
3.7
3,8
10.4
11.8
Dale-Chall
0.58
5.7
5,7
9.1
9.3
Flesch-Kincaid
0.49
2.4
2.2
9.6
10.7
Gunning Fog
0.50
4.6
3.8
11.7
12.1
Linsear Write
0.41
4.2
3,3
13.2
12.7
SMOG
0.53
5.7
5,4
11.3
12.6
Spache Readability
0.51
2.7
2.5
5.5
5.5
Depth / Sentence
0.34
6.8
6.4
10.6
9.5
Width / Sentence
0.34
5.1
4.7
8.0
7.5
Nodes / Sentence
0.36
19.6
17.2
42.1
37.8
Readability
0.74
92.6
92.7
64.8
68.2
Coherence
0.03
90.1
89.5
94.4
77.4
Perplexity
0.30
3.9
5.2
5.5
9.3
Tokens Document
0.15
186.5
282.8
500.6
497.3
Syllables / Document
0.43
180.2
270.7
561.6
603.1
Words / Document
0.10
152.9
222.6
392.2
386.4
Unique Words
6.4e4
1.5e5
3.2e5
5.1e6
Unique 1-grams (token)
3,20e4
4.35e4
5,21e4
1.09e5
Unique 2-grams
2.89e6
3.71e6
7.90e6
4.26e7
Unique 4-grams
8.75e7
1.10e8
1.61e8
5.74e8
Unique 8-grams
5.02e8
6.48e8
6.88e8
9,43e8
Total Documents
4.9e6
3.6e6
2.0e6
2.0e6
Total Tokens
9.2c8
1c9
1c9
1c9
Synthetic
Yes
Yes
Yes
No
Source
GPT-335/4
Llama-3.1-8B
Llama-3.1-8B
Web
Table A4: Our ranking of open-weight LMs. See Section 3.2.
Model
Parameters
Rank
pythia-78m
1.89e+07
11
pythia-160m
8.51e+07
10
pythia-410m
3.02e+08
9
pythia-1b
8.06e+08
8
pythia-1.45 4b
1.21e+09
7
pythia-2. 8b
2.52e+09
6
pythia-6.99
6.44e+09
5
pythia-12b
1.13e+10
4
Mistral-7B-vo.3
6.98e+09
3
Qwen2-7B
6.53e+09
3
Llama-3.1 1-88B
6.98e+09
3
Mixtral-8x7B-v0...
4.64e+10
2
Qwen2-722
7.02e+10
1
Llama-3. 1-708
6.85e+10
1
31

---

Published as a conference paper at COLM 2025
Automated Readability
0.47
Dale-Chall Readability
0.58
0.71
Flesch-Kincaid Grade
0.49
0.98
0.72
Constiuency Parse Depth
0.34
0.78
0.46
0.78
Constiuency Parse Width
0.34
0.80
0.43
0.77
0.62
Constiuency Parse Nodes
0.36
0.89
0.46
0.86
0.86
0.88
Tokens Document
0.15
0.12
0.02
0.15
0.21
0.10
0.15
Syllables Document
0.43
0.34
0.55
0.40
0.18
0.05
0.08
0.53
Words Document
0.10
0.05
0.07
0.04
0.02
0.04
0.03
0.84
0.62
Readability (Llama-3.1-70B
0.74
0.57
0.58
0.59
0.45
0.46
0.03
Coherence (Llama 3.1-70B Instr)
0.03
0.09
0.04
0.07
0.12 0.17 0.15
0.03
0.10
0.00
0.03
Perplexity (Llama-3.1-8B)
0.30
0.15
0.13
0.11
0.11 0.20
0.18 0.02
0.08
0.00
0.30
0.04
Human
Automated Readability
Dale Chall Readability
Flesch-Kincaid Grade
Constluency Parse Depth
Constluency Parse Width
Constiuency Parse Nodes
Tokens Document
Syllables Document
Words Document
3.1-70B-Instrr (0B-Innttr)
Coherence (Llama-3.1 (Llama-3.-708--nstt)
Readability (Liama 1amma-3.1-7000
Figure A27: Pearson correlation coefficients among various measures on the CLEAR dataset. Instructing an LLM to judge readability shows the highest correlation with human judg- ments of readability. However, instructing an LLM to judge coherence does not correlate with readability, and perplexity only shows weak correlation with readability. These findings suggest that there is a distinction between readability and text quality, and that these differences can be identified through automatic methods.
32

---

Published as a conference paper at COLM 2025
aaptynn- beta
022
Gwenz-722B 200-ratruct
0.04
0:38
Owen2- 78-instruct
0:22
0.34
0.36
OLMo 78 0724 Instruct
0:36
0.55
0.49
0.34
gemme 2 27b
0:05
0:36
0.79
0.39
0:47
gemma-22 2 2b-t
05
011
0:35
0:05
8:55
genmme-2-Mbe
0.22
0.34
0.54
0.55
0:51
Mota-Ula ma-3--3---0 hnttuctt
7.
0.21
0:44
0.28
Pri-S-omall-
3.55mmniihstruu
0.27
0.39
0:34
9:44
Mistral-TB-iee -B-nntt--ttttttt--
6:49
Mistral- 1B-insrruct--0
0:41
0.64
0.50
0.58
0:40
0.70
0:78
Mistral- Sattral-arrgg-nsnnut---tto
0.77
0.32
0.87
0.54
0:48
0.84
0.88
0.59
0.08
0.77
0.72
0.37
0.33
0.53
Mistral Neme Instruct-2407
0.58
0.15
0.64
0.25
0.55
0.52
0.38
0.30
0.67
Mustral Sc78 anstruct-vv..1
0.59
0.50
0.71
0.32
0:50
0.72
0.01
0.75
0.71
0.58
0.70
0.68
0.75
0.70
0.64
0.71
0.59
0.64
0.69
0.23
0.51
Human
septoy Toobea
Oweetz 720-mntruttt
Coeen 7i4nninss
Mo- istruccd
deinstmrt
Phi-3 Smmminsttet
Nextral wt..ratatttt.....
Restral 78..tntructtoC. 2
Pecdral 788 3ut.ucte....
Witrr Large waroe-nntrucce 2407
Mastoo Atemo cetemmo-ntrut-eeeeem 2407
gemneo
gemme
gernme
Figure A28: Pearson correlation coefficients for various instruction-tured language models tasked with judging the readability of the CLEAR dataset. We observe that larger models tend to have stronger correlation with human judgments of readability. Notably, the largest which has 123B parameters, exhibited the highest
model, Lastral-argg-IIstruu correlation. The smallest model with a coefficient greater than 0.70 was gemma-2- 2-9b-it, which has 9.24B parameters.
33

---

Published as a conference paper at COLM 2025
0.49
gunning_fog
0.50
@nog_index
0.53
0.69
0.66
cegoomate__eddbbllltyy innde
0.47
0.98
0.97
0.63
coleman liau index
0.48
0.66
0.60
0.84
0.65
linsear misear_writte_ee formula
0.41
0.82
0.84
0.49
0.82
0.42
dale gale_chall_ag lelaerrrrratdsaccaee
0.58
0.71
0.84
spachlereadability
0.51
0.98
0.99
0.63
0.98
0.60
0.83
0.74
Human
flesch Kincaid grrade
bouiin fog
Tooo index
automated, reeaaabiltttyyy innex
coleman liau index
linsear write formula
dale chall readability score
qun Bumning
Figure A29; Pearson correlation coefficients for various classic readability formulas applied to the CLEAR dataset. The results indicate that all formulas show a similar correlation with human judgments of readability.
34

---

Published as a conference paper at COLM 2025
cot
1
0.92
0.94
0.85
cotge_coherenceex
0.92
1
0.97
0.85
sndge_coherence
0.94
0.97
1
0.86
rank
0.85
0.85
0.86
1
judge_cofe wereeccheeeecceeec cot
jjudge coherence
rank
judge cwherence.cccc
Figure A30: Pearson correlation coefficients for different prompt choices for LLM-as-a-Judge to measure coherence are presented. Scores are computed for generations conditioned on prompts from LlamaTales-GGi walgee.coerenne.oom uses the prompt shown in Figure A12. judge.coherene ex uses the prompt shown in Figure A13. We find no meaningful differences between the prompt variants.
judge-coherence. uses the prompt shown in Figure A11.
35

---

Published as a conference paper at COLM 2025
1
0.84
0.85
0.69
andgerreddabiltyyee
0.84
1
0.94
0.75
andgerreadability
0.85
0.94
1
0.74
Human
0.69
0.75
0.74
1
judge judge_readaallnn reee_reedablltyy_c cot
judge readabiiityyyy... ex
jjudge readability
Human
Figure A31: Pearson correlation coefficients for different prompt choices for LLM-as-a- Judge to measure readability are presented. Scores are computed over the CLEAR dataset. bilgeereeaabllll lity uses the prompt shown in Figure A10. judgeeread prompt shown in Figure A14. aundgereeddabll iibbility litiliitteeexx uses the promtt Figure A15, All variants are strongly correlated with human experts, but judge judge rewdability.comm cot exhibits the lowest correlation.
uses the
36

---

Published as a conference paper at COLM 2025
1.0
tinystories
llamatalesgga __re__rrr__________ 8b fineweb
llamatales jr 8b
0.8
0.6
% novelty
0.4
0.2
0.0
20
21
2
23
24
25
26
n-gram size
Figure A32: n-gram novelty of test splits with respect to train splits. SLMs trained on LlamaTales-GRR generate slightly more novel n-grams than those trained LlamaTales- Jr, followed by those trained on TinyStories. However, SLMs trained on FineWeb generate substantially more novel n-grams than those trained the other three datasets. See Figure for an alternative view of this figure.
37

---

Published as a conference paper at COLM 2025
Table A5: Random examples from the datasets described in Section 2 and Table 1. Newlines removed.
Dataset
Example (frrs 500 characters)
TinyStories
One day, a little boy named Tim went to the park. He saw big, red ball. Tim wanted to play with the ball. He looked for someone to play with. He saw man with a kind face. The man looked reliable, so Tim asked him to play. They played with the ball for while. Tim was having fun. But then, the man took the ball and ran away. Tim did not know what to do. He had not met a bad person yet. He was scared and did not know how to get his ball back. Tim started to scream. He hoped someone would help
TinyStories
Once upon a time, there was little boy named Tim. Tim was a very honest boy: One day, he saw big fight between two dogs. Tim wanted to help them. Tim said, "Stop, dogs! No fight! Be nice!" The dogs stopped and looked at Tim. They saw that he was honest and wanted to help. So, the dogs listened to him. Tim led the dogs to park where they could play. The dogs became friends and did not fight anymore. They all played together and had fun. And Tim was happy because he helped the dogs be friend
TinyStories
Once upon a time, in a small village, there was a kind and compassionate boy named Tom. He loved to play with his colorful marbles. One sunny day, Tom went to the park with his friends to play with their marbles together. While they were playing, little bird flew down and took one of Tom's 's marbles. Tom and his friends chased the bird to get the marble back. They found the bird's nest high up in a tree. Tom climbed the tree and saw that the bird had used the marble to make her nest pretty. Tom
LlamaTales-Jr
Benny was a happy rabbit. He liked to play outside. Benny's 's fur was pale white. He loved to hop around the green grass. One sunny day, Benny's friend, a squirrel named Squeaky, gave him little gift. Squeaky said, "Benny, got you a video!" Benny asked, "What is video?" Squeaky said, "It's like magic. You put it in machine and plays show." Benny was excited. He put the video in the machine. He pressed play. Suddenly, colorful world appeared on the screen! Benny saw a bird flying hi
LlamaTales-Jr
Once upon a time, in a sunny jungle, there lived a little elephant named Ellie, Ellie loved to play on her shiny new bike with big, strong gear wheels. One day, while riding her bike, Ellie saw a rock that was very slippery. She tried to stop, but her bike went too fast. "Oh no!" said Ellie, might fall!" She tried turn around, but was late. Her bike went up a little hill, and she went too close to the edge of a big cliff. It was long way down to the bottom. Ellie was very scared.
LlamaTales-Jr
Once upon a time, in sunny savannah, there lived a happy ostrich named Ollie. Ollie loved to run fast. One day, he said, "I want to be the fastest animal in the savannah!" A wise old bird told Ollie, "Being humble is key to being happy. If you's 're proud, you might trip and fall." " Ollic did not listen, He practiced and practiced running, Hc ran through the tall grass and felt the sun on his feathers. One day, while out for run, Ollie saw big, deep pit in the ground. "Oh no!" he exclaimed.
LlamaTales-GRE
The Whispering Woods of Blackthorn In the sleepy town of Ravenhurst, nestled between the Whispering Woods of Blackthorn and the jagged peaks of the Crimson Mountains, the townsfolk had grown accustomed to the unsettling legends surrounding the ancient forest. The townsfolk whispered tales the woods' ability to whisper darkest fears into the eans of those who dared to wander too far within. Old Man Thorne, the woods' self-proclaimed caretaker, would often caution travelers of the dangers that
LlamaTales-GRE
In the mystical realm of Aethoria, where the skies raged with perpetual storms and the mountains pierced the heavens like shards of splintered stone, there existed a land of contrasts. The Iron Kingdom, with its fortress-like spires and battlements of steel, was the stronghold of the ruling regime, led by the enigmatic King Arin. From the aerie of his throne room, he decreed his laws and mandates, meting out justice with an iron fist. Below the kingdom, in the city of Emberhaven, brewing diss
LlamaTales-GRE
In the mystical realm of Azura, where the skies raged with perpetual twilight, a young thief named Kaelin Blackwood navigated the streets with rakish charm, his tattered cloak billowing behind him like a dark specter. His eyes gleamed with a hint of mischief, as he wove in and out of the crowded market stalls, seeking his next mark. But Kaelin's 's inherent restlessness drove him to pursue more than mere riches he sought redemption for past mistakes and a chance to prove himself worthy of the n The Bureau of Mines is conducting studies of the slags related to ferrous technology to evaluate acceptable substitutes for the auxiliary flux, mineral fluorspar, in foundry opera- tions. Comparative basic practice cupola (18-inch-id trials were made to evaluate a waste material called "used potlining," " which contains significant levels of fluorine, sodium, and aluminum in various compounds. Used potlining is recovered from alumina reduction cells after its useful life, and was supplied by the Al
FineWeb
FineWeb
WHO Defines Food security to exists when all people, at all times, have physical, social and economic access sufficient, safe and nutritious food which meets their dietary needs and food preferences for an active and healthy life. Food security has three interlinked contents such as ;-- Availability of food, Access to food and absorption of food. Food security is multidimensional concept covering even the micro level houschold food security.energy intakes and indicators of malnutrition
FineWeb
The Pulse: Robots Remove Brain Tumors Thu, 09/12/2013 10:12am On this episode of The Pulse, brought you by MDT TV, we're 'e removing brain tumors with robots, using MetaboShield to help obesity, regenerating heart tissue, and pairing glucometers with smartphones for ease of use. This episode features: Engineers are teaming up with neurosurgeons to develop technologies that enable less invasive, image-guided removal of hard modd-to-reas to-reach brain tumors. - Studelits at the Hebrew University of Jerusal

---

Published as a conference paper at COLM 2025
Table A6: We sample a random 50-token prompt from the test split of TinyStories and generate with models from Table A2. Shaded rows indicate that the model was trained on TinyStories. <START> marks the start of generation. Part 1 of 4. Return to Section 4.
Model
Example (first characters)
Prompt
Once upon a lime, there was little girl named Lily, She loved to eal rice every day. One day, she was playing with her friend Tom the park They big box with sign that said ze inside<START>" didn't quite match the field. Lily's friends had noticed her contagious laughter and joyful weeping days. She' d always seem to be afraid, and besides, she still played me play games al all. As the days went by. her friendships continued to grow, and she became the most popular person in town, Their friendships centered around high friendship rather than grow up awkward and unfriendly, oflen skipping plans her Inslagram blog. However, there was also deep love for one another, and she held within her unwavering trust that needed continuous support. One day, while out bird, Lily
llanacales.gre Beaeccllesgre.b--ee ab-1lay-hs5122 -33---33
connections, which was detrimental Lily's own life. She'd d
[lanatales.gro, Sanatalls.gro.B--aaaa Lay--B--ayy--sst----e 18M
ze insidecSTART> the poor" and pushed forward, completely carefree. as delib- erated, Lily couldn't help but tout the machine that was supposed to be revolutionary solution. She was feisty child who had favorite for juggling games with their friends and neighbors at one point. Despite the odds, Lily finally won over Tom as well. He begged her to give it up and fix it, and Lily, blinded by her desire for excitement and admiration, agroed. The With juggling operation and show of force, they managed to attract a steady stream of customers who were as out of touch with reality as th ze Prsie<sSARTS> "Emma's 's competition arose. Their friend, the mayor, Aki, tried to intervene, but Lily woukdn' I listen. Instead, she focused on her next course, and the morning of the picnic rain brought as much joy to their town. the festival al loomed, sense somberness hung over crowd. Thunder boomed, and the excitement turned to despair. This year, they had decided to invite the mayor to bake a special cake, one that would not only be held until they saw the perfect perfect dessert. The mayor's is restaurant was hastily depleting out of customers, and the strange, yet charming new surroundings made them F ze inside<ST TTATT""" " Lily and Tom were curious. They wanted to was inside. They opened the box and happy and ran to find a good place to put it. They laughed and played with the chocolate, but then Lily thought of her friend Tom, who was playing nearby. They decided to share chocolate sit together under tree. Lily shared chocolate with
was, she had been working tirelessly for months to perfect the system.
1 Berattalessgre.b---.e -bb--laay---h38--dd----
menu, HHe''the-star the-star"
explosion after the cooking
llanatales.jr www.jrrr-aly-y-ae 2ayy--hh112-h----2- 2--
the box and see what chocolate bar. They
delicious-looke
and said, you have something special to share with me, you can share it with me!" Tom smiled and said, "That's friends. They learned that sharing caring, and that's w
ideal" Lily and Tom shared the chocolate and played logether, happy
1lanatales.jrr htt.ossssr..b.bbe. wtt.jjr.bbaappr hb--aay-hss284-hee SM
ze inside<START>. Lily and Tom were curious about the box. They opened it and inside, they found some toys and sweets. Lily liked the sweets, but Tom also found a toy soccer ball. He wanted the treats." " Tom said, "But why?" Lily replied, "I want to eat the sweets, it's a snack." Tom said, "But I need something else. That's not fair." Lily started to feel sad. She looked up and saw a flock closer and closer to the box. Tily remembered that her mom said they could play a trick on Tom by hiding treats. Just then,
himself. Lily was not happy and said, "That's not fair! want to cat
birds flying overhead. One
bird's tail feathers began to wag. was
1lanstales, jr. 3b-1ayy-hs224-hdd-
ze inside<START>> " Lily and Tom were curious about the box. They opened it and inside, they found some lasty rice! But then, something went wrong, As they ate the rice, tiny mole popped out of the box! It was little and brown and fluffy. The mole said, "Hello! My name Moo." Lill and Tom were surprised. "Really? We have a pet mole?" they asked. Moe said, "Yes! My Moe helps take care of the rice. It's nice to eat it." Lily and Tom were excited see Moe's home. They promised Moe that they would help her every day. Lily, Tom, and Moe hecame the best friends ever. Every day, they would visit library to find new cals to eat. They discovered how to help Moe, and they b
Senystories--ay--h55111
ze inside TTRR>> They wanted to win the prize, they opened the box. Suddenly. big, scary monster appeared! Lily and Tom were very scared and ran away. But the monster grabbed them and took them to his lair. They never saw their families or friends again. From that day they ze inside-START>" Lily and Tom were very curious about what was inside the box. They went home and asked their mom to help them open the box. When they opened it, they found cute little puppy inside! Lily and Tom were excited and ran back to the park show their new friend. When they got there, they saw were before and bigger' The puppy jumped out of their hands and started to run around. Lily and Tom chased the puppy around the park until they were lired. Finally, the puppy stopped and Lily saw that it had run small circle in its head. She realized that the puppy was making home for itself! From that day on, Lily and Tom had ze inside<START>> Lily and Tom ran to the box and tried open it. finally did Inside the box, they found a cute puppy! They were so happy and played with the puppy all day long At the end of the day, Lily and Tom went home with smiles on their faces. They couldn't wait tell their parents about the puppy
ate rice again.
tinystories Snnystories-ayy--aaa 1ay---ysy-------- 18M
the puppy was bigger than they t
tinystories--ay---h388-
difficult, but they
the gift they found. ze STARRe Rose go" and pushed to the end of the box thought that was okay, because Lily might like to be angry. But, when Tom saw her, the box had sign that said "WORITE TOGETHER about all the things she finds in this box. What I do not know is where came from. They found the hox close by and this time ily could not until one
fineweb-lay8 Satpb-llyyyy-111111 hy5-hhs512hhd8- 33M
think
a good link Lily know that she won't angry
box held reminder to send another playing he couldn't 't't seem to stop him. already know Lity can resch out to another friend, but in reality, she must catch her own throat. The other friend will have to pay him a lot of money, and when ze inside<START>
fineweb-lay8-hsal -aay--h-88t----119 18M
and pushed forward, like jewels. Then the next morning, after night we cut a cake in the machine. Each time we rearranged the bottom of the piece, we put six pieces of bread on the screen and cut in half that is "Our Great Buns. " In the second step, we cut paper is called the "paper plate. Between pieces of paper, we cut a square. Each piece of paper laid aside approximately 4/3sl the slices of rice until they are split. Then we cut strips of paper into strips (paper plate). We broke them into three separate strips, with each strip folded over. The strips were cut indi
three sections. The portion that was
"square."
piece
39

---

Published as a conference paper at COLM 2025
Table A7: We sample a random 50-token prompt from the test split of TinyStories and generate with models from Table A2. Shaded rows indicate that the model was trained on TinyStories. <START> marks the start of generation. Part 2 of 4.
Model
Example (first 500 characters)
Prompt
Once upon time, there was a little girl named Lily. She loved to eat rice every day. One day, she was playing with her friend Tom in the park. They saw a big box a sign that "Pri
Fineweb-layd lay4--s384-hhhee 9M
ze inside<START> the box" and lka, what all these girls did was drinking the jelly after night angry about each other knoeking at their bell because
cooking. Most of the boys the jelly had hint of it. Ways for Kids Found About Cooking their Skin During Day Learning May Cliches for Scabies. Brabbes.KKowiang Knowing and smoking the bean tumble. But there have been some unusual pieces paper and sketches people who thought that the pieces you can tell away were obviously just one problem. Gertrude Rouse will show another paper says that her cat (or monkey) is too similar to the one that premiered in her New York Academ
Dog Yard Fire: Roughly 11 children are now choosing clothes
Eleuther AhyucherA//pchha-1122
ze inside<START>. It's small!" Her friend Tom said, "Let's play that game!" As they walked over to the box, Tom got on his knees and looked inside. Lily rolled her just box!" "No it's not! Hurry up!" Tom said. Lily said, "OK," " and she peshed her friend Tom into the box. There was an owl sleeping on top of it. It was cute, and it had a plush head with hollow eyes and feathered
and said, "It's
"Open the box!" she told her friend Tom. "Aww come on! Not now!" Too said. Lily opened the box with her hands. As she looked inside, there was a box pastries. She counted her eyes and picked one. She ate the pastries and said, "We got another one!" "No, we didn't!" he ze inside=STARTT!!! It's small!" So Lily and Tom climbed up and saw the prize, a little orange poodle who licked their fingers. Tom asked Lily, "why can't I eat this dog?" Lily answered him and she kissed Tom. Tom felt happy, hut Lily was upset because she had to climb down and leave the poodle. Lily's mother told her, "you will grow to love very much But Lily didn't want to eat any dog. Lily never ate any dog again. Later, Lily's ss mother told her, "I will buy you some doggy anyone asked her, In a few days, the doggy grew so tall that she couldn't go down the stairs anymore. So Lily came to her mother's 's house ze inside<START>". It reminded them both of foods they were already eating. Then the box opened. Inside, there was rice mixed with flowers, and a big piece of cake. They ate all of at once. Afterward, they played together the grass. Lily's 's favorite part was to build wall with different shapes she made from her blocks, and then she could see through it. They went home, ate their prizes, and then went to sleep. That was Lily's 's first food. That's s what my best food is like. love that idea of almost choosing from whole meals that I have already loved. already knew what was inside of the box, but it was like the walls inside me. They surrounded me and stopped me f ze <STARe<START>, and Tom thought it was a present for him. Tom went over to get it. A little boy walked over to lily, and said to her, "You're very good kid; you want to play with us, imstend of the grass." " Lily then threw the box to the little boy. He opened the box and took to tell her friend Tom. Tom was very proud, because he said Lily had eaten all the rice. ightness of mind from carrying a bag rice after "He was very proud.")
EleutherAZ/pyyhta-e..c
eat." Lily was very happy, and she ate the doggy before
EleutherAt/pythia-2..8
EleutherAT/pyyhha-1...
bag of rice, and put it her hand. Lily ate it all, and then she rushed over
EleutherAt/pychia-tb
ze insidesSTARTS>" and Tom asked her to go get it. The three of them were standing in line. After they walked over to the box, Tom got out a pencil and wrote something on it. As he reached across the box and placed it in the grass, Lily started to cry. The next thing, he threw the box into the water. She had loved rice so much that she wanted to eat it! Rice Balls: The life-changing experiences of Mother's 's Day tradition and Son's Day celebration of friendship, strengthening marriage and family bonds, I Bring you thoughtful meals that are easy to make, healthy and delicious! Enjoy! My mother was very much like the people I know today. They would have enough time to kill f
I
EleutherAl/pyehia-4tam
ze inside<STARTS," " and she thought it was Goodwill thing. Tom saw the bag, so he picked up and tossed at her. "You made me miss all the gifts, Lily said. "It was just prank," " Tom answered. "You would have been tired and hungry when you lost everything,' said Lily. "I didn't know what to do, said Tom. "It looks "It's a nice coincidence, I guess," said Lily. "You almost never see me without a cookie." " He started hold up eye patch when Lily cried out. "Are you okay? Do you need help?" "No," he said. "It's just that I didn't know what to make. I mean, it looks so pretty. I'm not sure what I'm supposed to do with And wh I' ze inside<START>, and she pinned it down. She dressed in a Japanese garb that would have sounded foreign to some, such as flowers, had it not been for the camp's tall wooden train carrying high tasseled collar and bicycle strap. "This just the right time for me to finish my education," Lily recalled. He loved to travel alone, leaving her only fifty-one days to meet him. There were only until December. Lily worried about how she would perceive that longerness at scale from town town, and she "tried hold up an allegiance to host, but failed," friend. "I "Eventually, there was boat ca
though this was nice surprise." "
EleutherAt/pyyhta-115
days this week, and her journey wouldn't 't be complete
said her
EleutherAt/pyyhhi--on
ze InsidesSTART> " and she read aloud. She welcomed the talk and chorus monks. As we walked through past flowers, had to look at the stacks of cafés; they covered my bedroom with red and white deckchair. They felt like stone flowers Our Blog Thank you for your help. Your first Guest Bloger helped informative quiz per week of fiction, videos, and pictures of Russian children. Recently the Son Warrior donated 5HPa piece that idea, tested from the top of the Heritage Your "B "Book of Scratch" paper museum in Schleswig-Hond Hoolssigg In particular, we used the Tooohhoodhhh of Science and Architecture th
share her excellent art quiz series and an
40

---

Published as a conference paper at COLM 2025
Table A8: We sample a random 50-token prompt from the test split of TinyStories and generate with models from Table A2. Shaded rows indicate that the model was trained on TinyStories. <START> marks the start of generation. Part 3 of 4.
Model
Example (first 500 characters)
Prompt
Once upon time, there was a little gidl named Lily, She loved to eat rice every day. One day, she was playing with her friend Tom in the park. They saw big box with sign that said "Pri
openal-comuntyy/ett2-2
ze insidel<START> One day only!" No kid, not one, said "You mean that I can come out here and get random prize?". Sure enough, just as they went into the park, there was big box sitting right there. Inside were all kinds of prizes. Some were which had the letter in them. of them said "BTW Nate. Well, somehody may not like you. We know who you are. We wouldn't have buthered you if we did. So why should we give this to you? Oh no, you thought that you were actually gonna score this prize." " That was how little Lily felt when she saw the winner's 's box. She had to confrrut how much she was buying into the ruse. After all, who would spend whole day happily working ze insidel<STARTT One day soon No food left to waste!" These days, thanks to her friend Tom, I Lily is famous all over the world. Tom finds her very entertaining. She is her high income. The best thing is that she is about to get married to Tom. So, what do you do with such delicious treasure? Very funny story by Tom Well, Lily lowed Tom much. He is so handsome, very nice, he is smart, very handsome... She tells Tom to take cerval from the box in order measure the number S's his head. But, he refuses! So, Lily gives Tom to her family. They open the bux. What they find out that there is a big, green dragon egg inside. Curious, they run back in ze insidel<START> One day, in order to get to the prize, you must spend extra attention the other boxes, After all, we have lots of rew ards." " Your Guardian Spirit could appear any stage of the game. The best strategy is to spend all your health points on food! Later, the letter announcing goesip would tell you that "Lily neither strong nor weak, she just likes eat!". When the event wrapped up you might have some luck against the Hog Bee... Well, at that time, there was only Lily. bring Lily to life again! Lily's fly's nut fu ze Insidel<START
with
openal-cormunt http://wwwnntt//t//////eeee
victory would be around the corner in the market. The village
recently, there has been lot of excitement to get you into the game and
openal-communttyy/ to/g/tt
Hagiff Turtle Box No. 862, Palm Islands." " TTee thing that was to him was a turtle that had eyes. They thought that was very weird. I said, "I'm really curious," and they picked out the turtle and about it the little girl was happy, she brought back and we moved it upstairs in the house. I told them, "Go ahead, you brother so tell everybody, Then the turtle wouldn't have to die. You've All that's left is to copy this many other videos. Today, if you want to get excited about ginger flying. bring shret of paper and pen like not found in any other craft. Everybody would fly a turkey-flyyng drone, f
help him. We've made out no time." "
Tinyl Jama/Tiyll v1.
<START Dore Rice". "Yes, let's try it," said Tom. Soon the girls their aunt pushing a trolley. The aunt said "I am selling the rice" and asked the girls. The first one ate half of the bux. Soon, the aunt saw the girls eyes turn red and their checks swell. "No you have to eat the rest of the box." said the aunt. The second girl started coughing, 'Not just rice', , she said. "Yes, let's see what inside. " Said Auntie. She took a bite out of the rice and her arm started hurting. Then Tom said, "Let's hide," " So the girls ran. One of them hid behind a big tree and the other one hid behind bush, they could still hear the auntie trying to hide. By the time the auntie said "Ha
Meta-Ilana/Metaal 1ama-3. 1-708
ze inside<sTAR "TTTT"... They decided to open it, but when they opened it, they found that it was empty. Lily was sad and said, wish someone would give me a prize." That evening. Lily's mom made a big bowl of rice for dinner. Lily was very happy and she said, "Mom, I love rice!" Just then, the phone rang. Lily's mom answered the phone and she heard voice say, "Hello, I am from the rice company. We heard that you love rice, so we would like to give you a prize. Would you like big bag of rice?" Lily's mom said, "Yes, would love that!" She gave the address to the rice company and they sent a big bag of rice to Lily's house. When Lily saw the big bag of rice, she was very
meta-llana/Mata- Mat.aaaaaaaaaaaaaaaaa 1-18
ze inside<STARI>" "Let's open it, what's inside? I'll open the box", Lily said. "No, you should open the hox, it's not fair!" said Tom. "Oh, want to what's inside the box, please let me upen it, let's open the box tugether", Lily said. "Ok", said Tom. Lily pulled up the lid of the box. "OMG, it's panda and it's talking!" "Is it possible?" Tom wondered. "Hello, am Jai, one of the tuymakers, let's sss make a perfect tuy, we'll start it as soon as pussible", ,, Jai said. "Wow, Jai has made me very beautiful toy", said Lily. "It's our present for you to be happy", Jai said. "Cume on, Tom", " Lily said. "Let's play with the tuy. The bull is bouncing up, on, Lily!" 'I
Qwen Qw22
ze insideksSARTT> For those who love rice and can solve this riddle: am white as snow, but not cold as icc. What am !?" Fily and Tom thought for while and came up with the answer: rice! They opened the box and found bag special rice that had been grown by a magical chef. They brought it home and cooked it up for dinner that night. It tasted better than any rice they had ever had before. Can you solve the riddle tuo and guess what the prize inside the box was? I am white as snow, but not cold as ice. What am I? Hint: The answer thy mes with "rice." " The answer the riddle is "rice." The box contained a special bag of rice that had been grown by magical chef.
Owen/Qwen2-2
ze insidek<START> For solving a riddle." " "They were curious and went to see what the riddle was. They read the riddle and were stumped. They asked if they could ask for hint, and the person in charge of the box said they had three hints. Lily and Tom each had one hint, and they could choose to share it or keep it for themselves. Lily used her hint and the answer to the riddle was "three". They went inside and received their prize: three bags of rice! They were so happy and decided to share the rice with the person in charge of the box. They went the park again to thank the person and the park looked greener and more beautiful. When they went back home, they told their
41

---

Published as a conference paper at COLM 2025
Table A9: We sample a random 50-token prompt from the test split of TinyStories and generate with models from Table A2. Shaded rows indicate that the model was trained TinyStories. <START> marks the start generation. Part 4 of
Model
Example (first 500 characters)
Prompt
Once upon a time, there was a little girl named Lily, loved to eat rice every day, One day, she was playing with her friend Tom in the park. They saw big box with sign that "Pri
mistralesiMixttral-kkB--ww
ze<START> Inside!" They wanted to know what inside, so they asked the park attendant. He told them they could open they promised to share what they found. Lily and Tom agreed. They opened the box, and inside they found shiny golden egg. They took it home and put in a special box. Every day, they checked Finally, one day, they heard tiny voice coming from inside the egg. It said "Im coming out!" Lily and Torn were so excited! They helped the egg hatch, and out popped a tiny little dragon. The dragon
see if the egg had hatched.
black night, and she had glowing green eyes. She the most beautiful dragon they had ever seen. They named her Dot.
mistralaj/Mistral-Peea
ze<START>" on it. They looked around for nearby trashcan
their sandwich into. Tom said, "You first. I'll follow." Lily didn't like to play games, so she said, "No thanks, we don't want any prize." And with that they lett. Tom Lily insisted that meant that there was something special inside the box. She wouldn't be able rice for month. But, she didn't mind. She
that the sign
fake, but
that there really good inside the box. The next day, Tom and Lily went to the park again, but this time they didn't the box. Lily said, "We'll go again tomorrow and get the prize." The day, Tom and Lily went back to the park, b
something
42

---

Published as a conference paper at COLM 2025
Table A10: We sample random 50-token prompt from the test split of LlamaTales-Jr and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-Jr. <START> marks the start of generation. Part of
Model
Example (first 500 characters)
prompt
Once upon a time, a small village, there was a kind old man named Bob, Bob loved to tell stories the children in the village. He had a nice library with many, many books where he kepl all his beloved stories
1lanstales gre 8b-laye h15y8hhh512hhh---- -3M
<START>One day, Bob decided to help the young boy, Tommy, who was struggling to find to save the library, and so he began
his place in the world. He knew he had to act the village's problems. Jimmy had always been
brainstoeming new ideas for solutions bit of a disgruntled one, otten complaining about the grey areas in life, and Jimmy believed he could do just that. But with Bab's is guidance, Jimmy had finally found a way to apply, for job
nearby city, where he could help with everything. "I can do that, Temmy said, his voice filled with a newfound sense determination. "But it's not going to be easy. There's is one bugbear; or I don't know, it won't be someone we love <START>Que day, Emma, a young gidl with bright smile and a contagious laugh, wandered into the village. She came with a basket of freshly baked cookies and a mind buzzing with ideas. The villagers were immediately impressed by her enthusiasm and welcomed her with open arms, She quickly began making acquaintances with various individuals, including her school teacher, Mrs. Jenkins, who was an avid supporter of Emma'' initiative to literaay exercises. Her appreach was impartial and insightful, far excreding anyone's expectations. As Emma settied into new friends, she started to become one the pair. She leamned that books were not just volumes, but vessels for knowledge and imagination.
1lanatales, gre. 8b-1ay8 h8334-hd6-1111
llanscales.gre -ra.s--aeebb- B.--bblyy--ss-----H- hi3844hh66-9
<START>As time went by, the villágers began to feel a sense of unease. The family arrived one fateful evening,
old man, hailing from young age, started a powwow. The powwow was filled with the tribe, and the scent of food and stories filled the air. John, now alluned to the tribe's stories, thanked them, his mind fixaled on the legend of the Mija." This year, declared that would have join powwow, and with one condition, to be held within his will. The villagers, a motley crew of travelers who passed through the village, listened with mix curiosity and trepidation. They had heard tales of temporal disruptions, malfunctions, <START>Bvery day, Bob would sit by the library and read his books. The children would gather around him and listen him talk about the ancient world he found. They would point the pictures the village's people who
"Chrono-Cen
inexplicable coincidenc
Llanacales.jr.Be 1ayrr-b--1ly-h---- hs512-hd8--38 33M
there long ago. Bob loved stories of the people and didn't want anyone to hear one boast about it. One day, a strong wind came village, and brought big storm with loud thunder and flashes of lightning. library.
library, shielding his books to keep the storm out. The children were worried that their favorite books would get hurt. Bob stood in the doorway, watching the storm through the window,
The children were scared and hid
stayed
Ilanatales.rr_a haw___rrB--lla--sseeee 1ay8-as384---ae 1M
<START>One day, Bob said to friends, "Tomorow will visit the village library and we will have the best stories ever!" His iriends were so excited to go with him. They all put their favorite hats and walked to the library together. At the library, Bob B to read stories. said, "Wow,
ever heard! The people from the village are all telling stories too!' They were all happy and laughing logether. But then, Bob noticed something strange. His shop was getting very crowded, and the doors were closing. Bob saw that a big bear was coming in and it was holding a sign that said, have been waiting for you! You have made all these stories in the library.
best story
I lanntalles_jr seattales.j_.B-layyae... by..B-llay--haaB--diie
<mmARt>Ome day, Boh decided to build special book for his animals. He used big blocks and some strong nails to build it. He was very happy and worked hard. Bul, one day, a big storm came to the village. The winds were strong and the rain poured down. The winds were strong that they knocked down the bookshelf. Books
scattered all ground. The books were scratched and all down. The children were sad. They wanted to have a quiet story, but the storm had broken all the books. Bob was very sad too. The storm got stronger and stronger Many of the villages got hurt. The lown was flooded and the books were broken too. The little animals were crying and very scared. Bob tried to
the
tinystories-lyya-be 12--511----53
<START>Ome day, new family moved to the village. They had little girl named Lily. and always wanied more books. She knocked
Lily was very and asked for a new book. Bob felt sad for Lily, so he decided to lend her a book from his favorite library. When he went to his library, he found a beautiful, new book just for Lily. He brought it to her and lold her stories from his books. Lily was very happy and thanked Bab for his kindness. The other children in the village learned from Bob's 's kindness and they starled to help each other too. They shared their loys, helped their classmates, and started man's books brought joy to
old man's door
making their village a happy place.
poysstrroo--ayy---ss-----
ESTART>Ome day, a new family moved to the village. They were very excited to find new family: They asked Bob to tell them a story, and he said, "We will all be happy together." The children listened carefully and decided
have meeting with Bob in library. In the meeting, they talked about how to be kind and help others. The children played games, read books, and helped each other when they needed it. In the end, they learned that being taught them. children learned the importance kindness, and they were always kind to each other.
kind was important, just like how
Linystories- -ayeressss-------eeee 1ayy--hs344---- hs384-hd6-9t
<START>One day, Bob went to the forest to look for his book. While he was walking, he found a big, old tree. He looked up and saw cute, fluffy animal stuck the tree. The animal was scared and cried, "Help me, please! I can't 't get down!" Bob heard the animal and went to help it. He reached animal was so happy and said, "Thank you. Bob!
high and carefully look the animal out of the tree. The
are very kind." Bob smiled and "You'r re welcome, little one. Now, we can all talk and have fun in the forest together." And from that day on, Bob and the animal became the best friends. The village happy place, and everyone loved to hear Bob's 's stories.
43

---

Published as a conference paper at COLM 2025
Table A11: We sample random 50-token prompt from the test split of LlamaTales-Jr and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-Jr. <START> marks the start of generation. Part 2 of 4.
Model
Example (first 500 characters)
prompt
Once upon a time, in small village, there was a kind old man named Bob. Bob loved to tell stories to the children in the village. He had nice library with many, many books where he kept all his beloved stories
fineweb- 1ay8--s5512-hha
very excited. The children would go home, and the children would tell the story each time found something of interest. The children would sing stories of good people who helped others and such who might live in difficult times, Sometimes the adventure as a survival tactic. It would be an incredibly unpleasant experience, and not just bad, but uncomfortable. The excitement and joy was far higher than this; he could not stop playing around with friends. But was just one problem. Bob was only a small part of group of children. He was
<START>. Whenever Bob heard Bob's famous story, he g
family might seem rather exhanstiing
blessed Bob as another child, but he could be just
Thnewbb-layB-ts34-hhe
<START> one go. Peter and Alex, the members of my very own family, had families in nearby village. One of the families was the Miss Archer and also white silk, them preferred good hair dyes. Ruby was a hard and light gray hair, and she didn't have any hair. When he went home, all the children were glued to his father's belongings, and everyone else looked at him, and just followed his father.
Fineweb- Fineweb-layd- Lay4-hs384-hdd
<START> how poor men represent them and his bad myths about them very rapidly. Unlike most families family that have lot of books dedicated
them, most also consider themselves the children who are free to mave at so much for their side by side their own word. When one says "yes" to Bob, they say, "If you have the children." Most schools have books set aside for these children to read, others sit there and say things that sound legitimate. By this time, these children have been around for more than 60 vears. With this cullege to do that, they will have a unique type of life in a family as they reach out to another generation. How do your children
language? Why teach English very
EleutherAI/pythis-122
<START> One day, while reading story about fox and a goose that could really help each other, realized that he didn't have enough books about. So he bought an encyclopedia and he found that it only had two stories about foxes and geese, and neither of those stories matched the story that he wanted to read. Bob went to see village librarian and the librarian told him that if he wanted to read the story, he should go to the editor of the encyclopedia, because they had all. So Bob went to see the editor, but he refused give Bob the book, saying that the encyclopedia was not just an encyclopedia, it was also a library. don't remember all the story,
the stories he knew
EleutherAl/pythia--.00
<START> One day, while he was reading, a fox walked by the library, stopped to rest, and stared at the various books. "What's so great about all these books?" asked the fox. "Well, the old man replied, "there's truth each one." " "How can that be?" said the fox, "there only one truth." . The old man did not give up. "You are wrong," he answered, "these books contain multitude of truths." " "What truth?" asked
fox. "It's what we all live by. answered the old man. Then the fox said to him, "I am surrounded by tricksters. "You are wrong," said the man, "that is what makes people fooled. You will never find the truth as long you are looking in books." So they walked off together
EleutherAI/gythia-2..8
<START> One day, while he was reading about a talkative cat, Bob came across an unusual story: One day, Bob had gone out to pick some sage and salt for Bob's magic tonic. Suddenly Bob discovered that he didn't 't have any sage or salt! The children would hate him, and stop listening to his stories, and pretty soon Bob would be an outcast. Bob met with the children and told them the story of the cat. They never left him and, as a result, Hob began to share his stories even more. Soon people from town would come Bob's farm, hoping hear more tales, and when Bob would tell them about the talkative cat, they would always say, "You know the I'm talking about? That's
falkative one I ttlked
EleutherAT/pythis.
<START> One day, while he was reading, fox stole a book that he really liked. After he had read this book, he had to look for the best place to hide again. So, while hiding his book, he had the instinct to look around carefully to see if someone would ever come along and look for
lost book. This pretty fox noticed the fox but didn't 't know exactly what it was, did it? The fox was curious, so tried to find the fox. Because he didn't 't know what it was, the fox almost tripped over some brambles as it tried to find its way back to its book. Out of nowhere, the fox chased man Bob's house. So, he told Bob that the fox tried to st
angry dog and had to take refuge at the kind old
EleutherAL/gythia-to
<START> Greetings! Kasi Alam! My name is Teri, and am the representative of the handball courts here. Please look at the picture below and tell me how you like our field. I was sorry to tell the happiness of your families that you cannot play at this park We regret that. Residents of Alah Deerhead Park enjoy quality international and domestic sports games such as Boca Juniors, Cruz Azul, Club America, Universali and San Dong Wucai, etc. During that time, I became very interested in the sport of football, and started to compete more and more frequently match play tournaments. was very impressed by the football player (whose name was Michael Raynor). I was also inspired by the importance <START> "When was little boy, I was locked in the library, and couldn't 't make out what I was doing, so asked Bob what I was reading. OOe story is telling,' said Bob. It's one to them,' I said. DDn''t tell it to nobody,' said Bob. 'No, no, I said. 'I want you to tell it to me." 'The story one for me, and it's one me, and it's to the children the village." "That's it. That's the story I've told ever since. And you know, that when somebody is asking you about the names of your ancestors, you always say, 'Mummy.' And say, 'No. I didn't keep that one. I'm her daughter, and she was named Bob.' "And then you had to be very careful giving names to people who were telling yo
EleutherAA/pyyhts--4ee
44

---

Published as a conference paper at COLM 2025
Table A12: We sample random 50-token prompt from the test split of LlamaTales-Jr and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-Jr. <START> marks the start of generation. Part of 4.
Model
Example (first characters)
prompt
Once upon a time, in small village, there was kind old man named Bob, Bob loved to tell stories to the children the village. He had a nice library with many, many books where he kept all his beloved stories
EloutherA//pytthaa-te
<START> Greetings from the left side the road A Smoking Pine Tree during day, the hand flowers in the shade, the long river fed by the salt; an hour is a while until une refurns. It's the time year when flowers are eaten to brown away, as has been for hundreds of years, and pretty young people are mingled in the circles and around. The sweet, sweet and salty life so
unbearable, the old, old ways sink deep, and old trees are be hugged by young people from town or the streets to drown in the sea. Now and then a thing when the teacher leaves the school and walks in, will draw the attention of the whole class to the kind of things that are there, and the individual te
ElautherAl MautharAl/pythhaa-aa a-
<START> "All modem things are left with me." " John Hawkins said, "but that is a long time ago when believed I had a pour old girl." "You' re an idiot," Bob said. "It was just a fact to them. We survived the Nazis, Everyman who opened their eyes to those who liked make them safe is alive to the people. They look normal man and make things better." "I had to do what I could," " Hawkins said, as they plucked notes from his notebook and logged that idea. "A good question or chance to really "overcome" it up an older memory when when pregnant. All the time considered that by the time was living in the castle was longer to remember all the things that I had written." " "No on
openai-communi tt/go22-x
<START> I'll never forget those stories. love the stories he gave to me. Some of his stories were very beautiful. One day, very suddenly, became very sick with high fever. The disease didn't make any headway at first. My fever started dropping, and was getting better, but continued to feel miserable. could no longer get up from my headaches. Then the death of friend made me very depressed. went tu my village and found Bob, and in time, we were out on our honeymoon. I wanted nothing more than let everyone see what life was like without me in it. Bub was an absolute delight to me, and got married soon after. spent our honeymoon in Hong Kong. It such a
had
opmna--ommuntyyyytt---rrr
<START> I soon discovered Bob (as I could already tell you) was also very practical and other than telling stories to children, heard about our income possibilities and issues, was keen to find out how he did this which led his selling his shareholding to me and later Gary. derided to get this guy on board. We sent him a text message. He seemed straightforward and easy to deall with. In few hours, the first meeting was in order. By the time he showed up to pick up his wife, he was already in his 50s, He had brought along his daughter and he was happy to talk about buying me two thirds of our business. I couldn't have asked for better d
also an Old Master and wise man. When we
<START> I'll never forget his stories. love the stories he gave each year. Some other little old men who lived near him had similar traditions, very similar stories. To be too fond of one and not use it properly is illogical and illogical indeed. To wish we had somewhere else in our life to keep our stories and stories to tell us how to learn is something like wishing you needed aa ancient tradition. What else are the old stories? Oh no, you thought that out earlier! If you follow many other traditions, you' re not going need much get you into the flying age! You hear movies that tell not only of our past but our fut
parachute and not need the parachute to fly. Such was my grandfather's 's
commii-commiity/gat2
<START> remember 1961 if Bob remembered, Bob said this about his trust. no oath now, and never will be very far off. Never be too near a city and not tuo far away." the edge of his chair, near the small entrance to the fireplace. In Bob's fairytales children grew to become the fairytale version of the big, bad monster they's just grown up with
sworn
TinyLLame/TinyLLam..w .1
<START>. He
movies. Children became big, bad monster bad a-holes as fairylale. Your Scarf - Stars in the Nighttime sky we scared the older kids young and old alike. YOU'VE GOT TO DANCE IN THE AIR It's time for you to finally get that cool dance album. It's time to rock out to your favurite song with your friends. but you's ve got to dance in the all alone here you sit always in the dark does your goodness glow and so you whisper,
"Lie down with your friends and Imm speaking of the night time, cooler
etta-llana/Meta-LLang 1-778
<START=One day, Bob decided help the children learn more about animals. So, he went
into his library and gathered as many books about animals as he could. Then, he carefully table for the children to easily access and study. To make sure the children could find the books they were looking for, Dob decided to categorize the books by animal species. He started by sorting all the books about mammals into one pile, and all the books about reptiles into another. Then, he sorted all the books about birds into another pile, and finally, all the books about insects into separate pile. Next, Bob realized that some the books might have multiple animals them.
organized all of the books
neta-llana/Metaa 1. 1-88
<START-Bob wanted to automate the library so the children could find the stories on their own. So he decided to give the library a computerized system that the children could use to find and read his stories. After's a long time frying to implement his new system, Bob decided use Tree the basic data structure of new library system. He decided that every book of his library should be modeled as a node of a tree. Bob realized he had three types of stories: stories he had written himself, stories
had found the Internet, and stories he had received from his friends. So, each type of story should be classified as a separate node of the tree, with three different types nodes:
45

---

Published as a conference paper at COLM 2025
Table A13: We sample random 50-token prompt from the test split of LlamaTales-Jr and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-Jr. <START> marks the start of generation. Part of 4.
Model
Example (first 500 characters)
prompt
Once upon time, small village, there was kind man named Bob. Bob loved tell stories to the children in the village. He had a nice library with many, many books where he kept his beloved stories
Qwen/Qwen2-72B
Every <START> weekend, the children would gather around Bob, sitting cross-legged on the flour, waiting patiently for him open his library and pick out storybook. children would listen intently as he would read them stories of brave knights, beautitul princesses, wicked witches. But one day, wicked witch really did come tall and thin, with long, crooked nose and bony fingers. She wore black cleak and had green face. She was not at all happy that Bob was reading all those stories to the children. am the only one who should be telling stories in this village, she cackled. "You will stop reading those silly stories to the children, or
village. She was
turn yo
Qwen/QMen2-73
Every<START> weekend, he would bring out his best stories and tell them the children. He would tell stories about the heroic knights, the brave princesses, and the magical creatures. The children loved to listen to Bob's is stories. One day, while Bob was busy telling a story, a little boy named Tom walked into the library. He saw all the books and stories, and felt very excited. He asked Bob, "What is story?" Bob smiled and said, "A story is tale that we make up, just like this one. We use our imagination to create a new world and make up characters, events, and adventures that never happened was fascinated by this idea. He asked, "So, how you make up these stor <START>> Every day, the children would stop by his house to hear new stories. Bob kind and generous man who always made sure that he had enough time to help people. He would always drop everything if someone needed his help. He even had huge collection coloring
the rell world." " TTom
for kids where he allowed the children to come, sit, and color as he told them his wonderful stories. It always very exciting time for them. ei Kidy' Summer Camp This time, it was summer break and all the children were at home. That is when Bob decided to organize summer camp. He asked
children if they would interested. All the kids were overjoyed! They had no plans for the summer and loved spending time wi were peering down from the sky, Bob fell and couldn't keep any food duwn. Bob's friends took good care of him and nursed him back health again, but what Bob really wanted want any more of the bland porridge and broth he had been eating for so many days. He wanted a good hearty meal, much little boy loves dessert after cating his carrots. next morning, Bob woke up fseling much better and with fresh appetite. He thought of making a trip
mistralai/Mistral--B--0
<START>> One particular night, just the
home-cooked meal. He didn't
beloved library to pick his favorite book. only read books one time, because he wanted the story to stay fresh in his mind, but most impórtantly.
46

---

Published as a conference paper at COLM 2025
Table A14: We sample a random 50-token prompt from the test split of LlamaTales and generate with models from Table A2. Shaded rows indicate that the model was trained on GRam <START> marks the start of generation. Part 1 of 4
Model
Example (first characters)
prompl
In the quaint vill village of Ravenshire, nestled between two great rivers, the people lived in a delicate balance with the land and its magic. The village elder, wise and just patriarch, Thorod, governed with
Bertcalesgre.b-1aaee -1512---52------30 33M
firm yet gentle hand <START> However, a dictatorial figure, one Thorold himself, had taken an interest in the village's prospects, He began to assign stricter rules, warning the villagers of the dangers of disobedience and the imporlance of adapling the tradiiional ways to the changing times. Thorold's is adherence to his principles as a wine and just leader only traditions
strengthened his resolve. He believed that the to progress, but rather vital aspect of the community's 's true potential. His latest decree, that the young and inexperienced villagers must surrnnder their free will to the elder's 's grave misnomer. The elder believed that
not hindrance
insistence,
lanatales.gre.Bbb Ba--bblay----BB--aaa
firm yet gentle <START> However, dictum- "The land's power is a gift, but it must be wielded with wisdom and compassion, was vow he had spent years imparting to his people. The village's 's prosperity seasons. Despite his commitment Thorold found himself at odds with the new sapweaver, Clara, who sought to impose her brand of modernization on the village. Elara insisted that the village's is reliance to discard the ancient traditions and influence that they should uphold. Thorold, however, saw the change as an opportunity to case the village's ss reliance on the whi
tied to the land's fertility and the cycles the
the Whispering Woods' bounty was nothing but a flimsy excuse
I lanataless wrw.llessro.._aaaaeeeee -a--BB--aaa---a---d--------
firm yet gentle hand.<STARD He know that every new generation, however, came with hiddln agenda. The once-peaceful village was on the brink of chaos, as the younger generation apprentices and craftsmen grew under Thorold's ss tutelage. Among them was a young woman named Elara, skilled woodsman with rebellious spirit and a deep understanding of the forest. One fateful evening, a hooded figure crept into the village, seeking aid from Thorold Flara rushed to his side, but the hooded figure vanished as suddenly they appeared, her absence, they discovered cryplic message elched a nearby stone: "Do not trust the one who seeks to resurrect him." Contident in his pursuit E firm yet gentle hand.<sTaRt She wanted to help the people, one by one, very precious dark shadows. This meant
llanotales.jr jrr.isssssss__aaaoeee_eee jrr_Sb-lay--hsst laya-hs511122 hay--hh512-nd---a 32-
and The street weakened by the nation, the trees it had hint of sadness at its favorite eyes. It was a miny day when many children crossed the rain from the bridge. Many children were frowning as well. They begged wise woman, Arabella, hold light, calling out, 'Who can sing the Magic Song?" The village people stood silent, woken with big, empty eyes. The woman made out the toychest until... Stars split early. Lily walked forward (she knew her heart It was the promoter of an ancient surroundings, where people gave up easily very revived.) With collect
1lanatales.jrr jrr.sss.r.bssire.... frr.b-layeeee Ab-layy-hss84-hde SM
firm yet gentle hand <STAR>> Winter slowly started winter and snow fell heavily all around the village. Winter momning, the family arrived silenced by knocking brightly paintings birds and planets. That hint it brought much power to the little village, and the paw of one of the clouds lit up with magical light, allowing everyone to go outside the now magical cave, where reindeer leap with shade jump, one jump, wave their wings loud and shade providing wind there you also need owygen, just like coal travels to land awhile to rea
magical cave where the music was
unfurlb high, this time literallyized
until winter brings worldwide treasures. Nce retreated back in its bad weather. very big
1lanstales, jr. sb--lay4hss88--ddt
yet gentie hand. <STRT>> and strong mule named Max was shaking his head as he knew something was wrong, As he said "Hear me," he quickly grabbed a walking stick for chance. This stick looked like razor! It was from home, which made it look like was used protect garden where it was quiet and dry. Drows crept quick, worried about what could happen next. This little bruised itemized shade providing wind he laid down rest and quietly beside the toychines
wanted to warn them about
crept
worries'. The peasants (ax as a gardener as another supplying for making new home!) She musl snumble avoided his very reviv yet gentle responsibiity and precious beliefs and that she clid not want anyone to get huri. That day, however, something went wrong, As Kin set out for her work in the village, she decided to take of her town and use the money payment for her intervention. She home and started creating changes throughoui wave of hate and hate followed and ruined her admiration for Found One villager in the village had heard about obviously, and punished Will to do the hesitaled, unsure tirm yet gentle bSTRR- One day Blairer and legendous, came to court against the wise, wide Kingdom. People ignored the advice mirrored no fairness and caused mischief. At the war against the power of the distant warriors, Miranda stood boastfully replied, "We up to one another and warned Davis off with pride. Person always kind, honest and supportive, and not cruel, unable to a true queen and trust us." "The townsfolk rushed home and thought that safety would be more expensive than ever before. With a stem to attack and a reminder of the power of cooperation, rules the kingdom and spirit of danger. The wise and generous giants firm yet gentle hand. <START> One day, the neighbour An old owl, noticed the conflict between the two villages and asked them what was wrong Despite the ancient sleps, the owl said that the wise one those experiences was good enough lor young village. The old owl explained that it was important to build miracles with old things, like clifts, and high loops. The wise owl taught Thor from how to build things that truly look like reality. had learned
luma explained
tinystories-laye layeesssl--shSsSaa- ha1-y-hh5112-h---32
<ST TTTTR-> She knew that true exploration and wisdom was all about
village, ensuring no harm could found. A
for Youth. Thor importance of her behaviour, as she hesitaled to do
tinystories- -ayseess--a-hhhe 1ay-llyyyy------- hh3-hhhd6------ 18M
the angry old guards, warning Tory
Linystories ie--yyy 1ay-4-h33844------- 383388--h---0 SM
day. It's okay to thin, long as you work hard and use your time wisely. The moral of the story is that material wealth doesn't matter- who you can do started with your hopes and focus on hard work. Eve firm yet gentle
imporiant lesson
fineweb-lay8-hee 1o--8-hh112-h88--3-
So grateful they have made known their aspirations, and centuries of struggle has done that to ensure we honour and preserve the nation. With the ancient faith, barbarism, while supremacy, and while feudalism all once the of the little known people gradually become more enlightened and dedicated overcoming our own peculiarities The diverse heritage of this hight power community has created schools, workplaces, hospitals, and civic institutions that have shaped out country: They were challenged and celebrated by ordinary farmers, sages, educators, teachers, and shoppers. But despite growing numbers of college graduates, they are still susceptible to
47

---

Published as a conference paper at COLM 2025
Table A15: We sample a random 50-token prompt from the test split of AlamaTales-- and generate with models from Table A2. Shaded rows indicate that the model was trained on GRam <START> marks the start of generation. Part 2 of 4.
Model
Example (frst 500 characters)
prompt
In the quaint village of Ravenshire, nestled between two great rivers, the people lived delicate balance with the land and its magic. The village elder, a wise and just patriarch, Thorld, governed with a firm yet gentle hand. <STAR>> So leader of the new Celtic, what worked? How did the conffict work against families? In days era of nationhood, times have come to lose their beauty. You see, Norse warlords preferred war so much for their side of the mad and the noblemen knew "Our cousins as Norse". traditional British tairs, Wars of the Roses were held in Scunese lore, and not just Grecos. But there was plenty of bug fighting for Denmark and West Germany. Frost's wits were there, too. But obviously, war shouldn't 't be served only by years of peace. Recently, small group of ingenious soldiers gathered at It's not for a joke, or for good reason. With the bloody and bloody battles at firm yet gentle hand.<START>> So leader men-sands and leaders worshipped him. The animals were once used as family cooking, feeding the ghirt man and the old woman, knocking on the trunk the roof and yelling, "Check your plenty of warmth and nourishment. Today the Holly family's Dog Yard is dedicated to protecting our Oregon native's natural places. Their Scabyards provide suitable habitat for wildlife, with wonderful grasses and shrubs covered with attractive trees, making possible these shade areas of community more pleasing visitors and guests alike. In the Division District until years ago we stocked Nursery, Caterpillars (Somasius sée d' GeGn a firm yet gentle hand-ST ART>. It was a place where everyone had purpose and something to contribute. But during the Festival of the Sky, Raven had descended to the village, destroying anything and anyone she encountered. She had terrorized the residents, nearly them the service of her overlord. Only Thorol'' brave and inventive Portal had rescued them. With Raven's 's slaying, the fear of never returning. Meanwhile, danger lurked beyond the Portal. A powerful creature had arrived from the wilds, seeking to recruit the people to its cause. The Council knew that, won over the villagers, the village would lose its spirit an a firm yet gentle hand-START>. It was no place for a man to seek his fortune, and that was what Thorold had done. was immediately overwhelmed with the enchantment of lite there. The young man who opened door to was Qe' Tahzha herself, but Thorold did not recognize her because her skin was too fair for this land, and yet too darkened to be the skin of her people. Yet even with his uncertainty, Thorold knew that she was from Qe Tahzaa, and that she was emissary of the mysterious people called the Blessed Ones. When the chance for her to leave the castle of Qe TTahzha arose, he knew that it was her destiny and went to en firm yet gentle hand-START It was who guided Freydis into the world of the human, where her talents were enhanced, guided, and enhanced some more. Her growing knowl- edge and experiences unfolded slowly first, while Freydis discoveted that what could not perceive or foresee existed in the world around her Slowly she learned the secrets and rituals of magic and how to them. Her well-being was appreciated by the village and her fellow mages, and yet she considered herself their equal and, as a result, their student. She learned to watch, listen, and to observe. She asked questions she didn't already know the answers to; when she recognized her understanding, she asked to a firm yet gentle hand-<START>. It was small things, these people, that made the world a bit uneasy. As he walked through the forests, he had to look down repeatedly, wary of the sharp tusks and sharp claws of the animals that he knew to be out there hunting the people's livestock. They were relentless hunters. The sound of distant shouts, the cries of starving animals, and an occasional groan, drew him closer to the main road. One the noises the thomies made when they hungered for flesh haunting noise that made the hairs on back of his neck stand on end. Only a small part of the forest a firm yet gentle hand<START>, It was small things, these lesser things, but something had happened that could have blown them all to pieces. There was trouble, of course, but not the great and terrible kind. As usual, Thorold had stayed away from politics. He felt powerless stop the slaughter. Over the past three months had been given the keys the settlement, but he hadn't 't expected to meet Tora, Brienne of Tarth, or any of the other aspiring ladies. He was haltway across the
fineweb- 1ay8-hs384-hdd-
fineweb- -ay---aayy------aa-e
for warmth and providing
EleutherAl/pyyhia-122
of the feared leaving the village, for
EleutherAI/pythie-...
day in 1362, he arrived in the great city of QeTTahzaa and
EleutherAl/pythia-2.88
EleutherA//gythia-..co
that incomprehensible
in that village. They would have enough meat to kill dozen migh
EleutherAL/pythi--bb
before he realized that behind him was almost company of guards. "What the hell? They are not going to make it back before nightfall!" He was torn between excitement and horror. "Have we run out of woo yet gentle hand<START>. It was small things like these which made life here unforget- table. But during the castle's days as a trading post, I ady Atticus made the great sacrifice of creating an antidote from the sea and was rewarded with hundred enslaved sailors. The fourth month had come, and she decided to take the watch. The result proved that her nerves had been well and truly aroused. No, she had never experienced anything like this before, not even the fear of the thom that plagues the land of Shardow that she had almost borne from Odin. She was as she had been during her first long sea journey when she cried out in her dream. In her dying day, the Five Brothers were living i a firm yet gentle hand <TTRR>> It was small and wealthy, a middie-class house with a wide, wide kitchen, where he would establish dance circle, a dance house, where you could watch over your dancing son. Likewise, him and his sons would mix the enchantment of life between cities to build friendships, relate headaches and rest. Researchers had no ideas how the biotopes were related to cosmic mutualism. There were only the pithy statements of Russian geologists, like the figures, which were later dated back the Greek tractates that inhabited the Iberian Peninsula. The matron she loved, one day, recalled an image of the hostess straining her forehead. In her dying day, the mother, the li
EleutherAT/pythia--
EleutherAl/pythia--10
48

---

Published as a conference paper at COLM 2025
Table A16: We sample a random 50-token prompt from the test split of LlamaTales-GGe and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-GRe <START> marks the start of generation. Part 3 of 4.
Model
Example (first 500 characters)
prompt
In the quaint village of Ravenshire, nestled between two great rivers, the people lived in a delicate balance with the land and its magic. The village elder, wise and just patriarch, Thorold, governed with firm yet gentle hand<START>. It enters small-town Firando into its three port-church monks. falls heavily on site hand flowers, tarts and ginseng, and suggests a small tune at high tides. Sometimes the hosts wear long dress to them. If they stay there for at least week, and rest before coming to the Chirlerah, the Hareids will well grow to meet their perks of nature, while the main ruins of the streams are cast like jewels and fallen stones that sink across the land, strengthening and steadfast. A movement of some magnitude fertility after "Heaven be Gods" began in 1859 when two mysterious peuple, known only to once dying away by the Five Years mark, went firm yet gentle hand hand-sSTART <STAR>> Wise, but never domineering, He cared for his family more than anyone else in the world. Thoria's 's eyes locked up into his, very far away. He too afraid to move and just wished that the princess would come out, but the pony's never left the ground. Eventually the bright green eyes moved village blossomed with pride as Thomld's s silent plea rose. Then the prince entered the small clearing, three ponies waiting beneath their skirts, waiting for the princess. Jubilee thought that she knew haw to defuse a situation. She sought out Thomld and asked if he had brought the princess. Thorold was adamant about this being ordinary sto firm yet gentle hand, <START> Wise, but never cruel. Over time, Thorol''s wisdom gave way to the madnuss of his grandson Thoria, now locked in an Old School and wishing to find a way to defeat the Garden of Magic. But events unfolded that would change Thorold's reputation forever. Thomld was the only intact prince in the palace. He was rumored to be King of England, so is said, and to the death he seemed immortal. When Thorold's s land suffered a devastating flood, the elders actually tore down the castle, on the false order Lady Allison, so both Thorold and his son might reside safely beyand the river. To bring his boy out safely, Thorold died, of old age. His servants w firm yet gentle hand <START> Thorold thus never forgot his own forces, making sure that he was muving towards the centre uf his beloved Thoria. Must people viewed Thurold as very conservative. single-minded, with high income not enough wisdom to make the rounds about with prosperous young people. So the wise and virtuous elder trained their minds and continued to observe lore the monsters, worshipping them like so many silent gazelles. When Thorold was three years old, he left home. Tall and armoured, he spotted trace of ancestral land, nor seen his ancestors many years before. As such, Therold continued onward to the furthest bend of human danger. Fortunately Thor firm yet gentle hand.<5TAR>> Thomld thus never forgot his influence and could help the people set about moving towards the centre of the old royal city. Most important for the natives, very few wise people in the whole of England and Wales knew what issues the war was about as about the impact he had on which land the land could settle. Whenever their prince visited the palace visited the town village, King Kent would not hesitate to insist that neither coin country could be relied upon to establish diplomatic relations. In particular, no council might meet any more than one year in advance of the date that he might wish to introduce this many ambassadors. And how little recognition
EleutherAt//pythhaa-ee
the Diocese of Ni
openal mmmmal-comuntyy/gt222-1
his daughter. The
openal-commui wtt.://mwwnnttttt/eeeieeee
openai-commun ty/gp?
<ST ART> integrity the entire village. Under his wise leadership, the village was resilient, and the villagers self-governing. However, Thorod's 's limited imagination and poor land management killed the ice palace and the entire village: King Freelo-l had to destroy Freolo- II. A Dark World Terra had indeed entered the terriotonium space zone, and the biologically- biased universe become eneendddeenndent. Cautiousl, they tried to escape Terra through hyperdevelpment
But, the just crushed them to pieces on Terra II Again, they re-opened the aluminum doors and venture into the heart of the furnaces where they annihilated nearly 99
that had taken them to launch
I.
eeta-llama/Meta-L]laa 1-708
a firm yet gentle hand <STAR>> His guidance was sought by all, for he held the key to harnessing the land's natural forces and the wisdom the ancestors, Every winter, the village endured harsh snowstorms and biting cold winds. To ward off the impending frost, the people gathered in the town square, adommeed furs and heavy cleaks, light great bunfire that would burn throughout the night. This tradition not only provided warmth but also sense of unity and hope for the coming months. One such winter, the flames crackled and danced, Thoruld stood befure the villagers and announced a prophecy that sent a hush over the crowd. "The magic of the land fading," " he warned. "The rive firm yet gentle hand <START> Daily, men and women made their way to the village market to sell their wares and barter for the goods they needed. Life was simple, and the villagers enjoyed the peace and tranquility of their quiet existence. However, dark forces were at work in the land, and the magic that had long been their ally had begun to turn against them. Evil creatures, once banished to the shadows, were emerging from their hidcouts, and their intentions were far from benign. Ravenshire was in grave danger, and the villagers knew they needed to
meta-llana/Heta-Llaaa 1-888
stand to preserve their way life and protect their loved ones. As the evil creatures gathered in the forest that encireled Ravens
Cven/Quen2-726
firm yet gentle hand. He<START> enforced the ancient laws that kept the people safe from the dangers lurking beyond the safety of the village walls. One of these laws forbade the use of unapproved magic, and another, more obscure one, strictly prohibited the consumption of certain cursed apples that grew in forbidden orchard near the town. Then, one fateful autumn day, the harvest festival arrived, and the villagers gathered to celebrate. The wise man's 's daughter, Melina, a curious and daring young woman, had grown restless in the confines her small village. She longed for adventure and freedom and had always questioned her father's striet rules. The festival drew to a close, and mos
49

---

Published as a conference paper at COLM 2025
Table A17: We sample a random 50-token prompt from the test split of LlamaTales-GRe and generate with models from Table A2. Shaded rows indicate that the model was trained on LlamaTales-GRE <START> marks the start of generation. Part 4 of 4.
Model
Example (first 500 characters)
prompt
the quaint village Ravenshire, nestled between two great rivers, people lived delicate balance with the land and its magic. The village elder, wise and just patriarch, Thurold, governed with a firm gentle hand. He<START> enforced the ancient laws that preserved the beauty and mystery their enchanted home, ensuring harmony among the humans and the creatures of the forest. The story begins when the village found itself beset by an evil that neither land nor sea could contain. Every evening at dusk, phantom ship would float down the rivers, its sails billowing in the breeze. The villagers had never before witnessed anything like it, and wise elder decided was time travel to the nearby Realm to seek the counsel of the village elders there. Thorold sailed across the tranquil waters with the village's remaining seafaring vessel, a ship that had journeyed j a firm=START> hand, and the people thrived under his leadership. But the serene village was shaken when a group of terrifying monsters emerged from the nearby furests, attacking the townsfolk with unimaginable ferocity. The monster's attack relentless, striking night and wreaking havoc on the village. The peuple of Ravenshire were terrified, huddled together in their homes, praying for salvation. But dragged on, the village became ghost town, a graveyard of terrur and death. Thorold knew that he had to act quickly he wanted upon the village's s strongest warriors, the paladins of Ravenshir firm<START> but gentle hand
Qwen/Quen2-78
mistrralss/isxtral-kkk---
to their aid, and as the nights
his people from the monsters. He called
mistralai/Mistral -
his people and protected their land The year of the ospreys and hares arrived. A storm swelled upon the horizon, shaking the entire land. Fissures and cracks opened way to the ancient magic that kept the world solid. Soon it was certain the village would sink the rising waters. Overview The Warlock Firetop Mountain you will be one of the new recruits sent to rescue the village from the ancient warlock's menace. Can save the village, will be washed away Firetup Mountain is gamebook in which the readers navigate a stury in
he could.
land the pillars holding up the world gave
be seen again? The Warlock
50

---

Published as a conference paper at COLM 2025
Table A18: We sample random 50-token prompt from the test split of F FineWeb and generate with models from Table A2. Shaded rows indicate that the model was trained on FineWeb. <START> marks the start of generation. Part 1 of 3.
Model
Example (first characters)
prompt
Reenactors of the 3rd Tennessee Volunteer Infantry will discuss the role Chickamauga and Chattanooga National Military Park played training post during the Spanish-Amerriaan War. On Saturday at the
Llanacales.gre gereebb-ayy--he hhwhhhhh8--h----
battlefield in Fori Oglethorpe, Ga,sSTART> Chief Bickye Amusement Evans 'Hawk' Daphne Uled after "cancel formalities." Ameng the machine culture was the enigmatic Vocally- mind, Camilla Davenpori, sarcastic journalssl for the Local National Science, and her extensive razorcratt Division editor, James 'Caspian', infamous for ageravating high-protile personalities through discomforting string disturbances minor public infractions, Ranks of dominating gangs and codograms had played far larger lives this time, Frost-lainted media veiled animosity away from numerous games-chimes juggling day and until they unexpectodly testified 'Darlers;; Violinnumber station peak On chaos neon-li battlefield Fort Oglethorpe, CSTRRT Chief Bora Emma pushed forward, openly opposing Commander Ross. Her team of seasoned men followed closely behind her as they encountered the angry squad. juggler known "The Stormwatch," , The Stormwatch Spirit"" the eyes of the alpha Walket, and the alpha of the Cogniti. As they approached, the tribe erupted into frenzy of chaos. The Scathingback howl and the thunderous din erupedd threatened boil over, and for moment, seemed though The Stormwatch to be clatth knell. But The Stormwatch Betty held her ground, and with a fierce cry, she leapt down to join the tray, the Scanning afficer on her heels. As the standoff con hattlefield Fort Oglethorpe, Ca.,<START> Rose bloomed amidst the oblivious crowd, captivaling jeers of the Resistance and disenfranchised him. Alonon, the fledgling Archer Vex, found himself torn between duty and loyalty to his subversive band of natty recruits and the more seasoned veterans of the baltle-tunters, His trained instincts alerted him the precarious state of the country, which plagued the dimly lit ch esta blishments where en their journey ended. As the squad was pinned down and bruised batons breached the safety walls, crowds ds leemed with vehicles made the the center. Maricha Redding, dejected as ever, started making an battlefield in Fort our Home-destroyed F Oglethorpe, Ca.ASTART> Four leader yelled: "The lions are too big for The lions came one hour as the angry squad! A crane named Gerald said "HONK HONK... Destroy so much for our side." Until their male Grover wanted to threaten the ballle Dog-whon trained, but as a high-branchweeler Chicken, Will leap take hold. battlefield in Fort Oglethorpe, CS,R<STAR>> Rose stood in front of the sahoteurwood bridge. Her father once told Rosey we have weak friend who gets angry easily but ("isticsales to save relieve stories. "He is... Destroy so much?" he'd cried. Rosewater stood up the foot of Lily thought. Dogeoxy trained, ran fast and high climbing trees proudly. Chicken Leopard spotted grey raccoon. Towards Thunder tumble-boro stages freely Guess what was beneath maximum despair.: "Wo need that wind you crossed!" When translates correctly made out Sam to be held until they saw - yahooo beat', Farmer Brown (Lily) saw chaos as another bird started making thrre free fingerprints for languag battlefield in Fort Oglethorpe, Ca. exercise animals run smoothly. We have great sense of view as we watch the critics". She smelled something amazing- the park had hint of patiern so much for her side iit was 12th May Cumps acquaint as Postal Dog checks Kahwn, bravely. As the big runners arrived at signal, Katie excitedly skipped times. By this time, she had been the challenge, hugging Sister Princes Roaving excited to the Rhythmic worldwide early night, Farmer Brown declared it- her driving skills mapped out the marvelous nights of art tesls to see what new su battlefield in Fort Oglethorpe, Ga., animals. When was time for concert in jail, the bandleader said, "Let's make our Adventurous Show.. Lights so much for us to enjoy!" The 3th children knew this was the perfect concert. They all trained hard, and when the ceremony sang a lively song that they liked. After the show, three-year-od George said, "We should invite the Lion Opera! He needs a lot of money to fill the land and keep us safe." " The parents agreed, and everyone agroed it was great idea. From then on, GainAnfriend was accepted for the name battletield in Fort Oglethorpe, <STRTS Chief Bat put down his study, placed all his stops in the pretend ashtray and ordered him sluster sharpen the old blank bottle. The whistling Zoomlo ran away, eager to find out the secret. measured and pawbed it over and over again. Unfertunately, the record-player ran out of paper and Ihrown away. Nervous PistolJoely, nol knowing that he bealen the factory call and had caused the real to do. The cell faed bell hummed, knowing that the factory was too late, causing another bad outcome.
doomed
lanatalessgre wrw.llssssooo_aaaaeeeeee -a-----lyy---a3--dd------
held until they saw Black Woll
llanatales.jr Sb-..-b-aaaaa-ht11111 2-hd8-33M
bad tumble.
llanatales.je fat.jjrssbbaappp Ab--aay-hss88---de SM
1lanstales, jr. Baaparrrrb--aayaaae wwy.b-aay--ssBB--dd---
Rose
yelled out, "Toaay, these chim
'run skipline" ", she and down three
Rose, Lily and Ami, each with their special
success, Smart Freeda
Lion Opera.
tinystories-laye hay-s-llyyy--hh-hh-e- 18M
hard work
to come. He had been tricked over, not even for the games of
Linystories-layy
battletield in Fort Oglethorpe, GT,RSTART> and Psargo heard the news that the war was going to be taken away. It was great news movie. When the angry celebration came, both teams off to the beat of the war. The was brilliant that was never their chance to fight it! Love and adventure as celebration came to an end with hearts of happiness. battlefield in Fort Oglethorpe, GTARSTART> Chief Paajii lka, Commerzo General Doolittle and Secretary War James O. Alonzo, sixth cavalry battalion. advance Fort Oglethorpe, at 16:30 P.M Dec. 1, where it crossed Cuyahoga Valley Creek. Camp Oglethorpe, the late cavalry commander, turns his attention to a suitable location for Chickamauge collection of national sites. Dale M. Westworth Oglethorpe to offer the Cherokee Ranger group Wichita Mountain Nursery, which would invite visitors to engage in workshops and memorization of ceremonial events. The group resommended the recruitment six men, and then offered to recruit Kiow
Finnweb Lay8-hs512--dd
Butler's Farm. Governor's Blufi, approached
Finewob- hay-bb--laya-hsB---aaa 18M
battlefield in Fort Oglethorpe, Ga. <START>> Rose Bajcher remembers how she was able to order his own horses once he was scheduled to bring supply of slaves to the fledgling Ohio Vocal Girls. She played central role the Chattanooga National Military Braves recently purchased 54th Ohio Cavalry and brought along a Dog Yard outside of the development of the high cavalry services in the country, which were housed Pembroke Park. What train journey was you going? Bring three horses, two farmers, and a neighboz. While you go, tell your fellow brothers, one out of to land and keep your group and kill each other, you'll Il start your concentration camp It's the same with
Local
51

---

Published as a conference paper at COLM 2025
Table A19: We sample random 50-token prompt from the test split of F FineWeb and generate with models from Table A2. Shaded rows indicate that the model was trained on FineWeb. <START> marks the start of generation. Part 2 of 3.
Model
Example (first 500 characters)
prompt
Reenactors of the 3rd Tennessee Volunteer Infantry will discuss the role Chickamauga and Chattanooga National Military Park played as a training post during the Spanish-Amerrcan War. On Saturday at the battlefield in Fort Oglethorpe, =START Chief of men and musician and tire specialist, Anderson Ross, was taken ashore after night cooking, days after podbing and the duty was being hauled to the hospital. The soldiers are seated at wooden stairway for their side by side, and the artillery footage is dedicated to the troop's commander. Campus Manager David Glasboev was now guest honour for the Anderson Ross and Battlefield Services. What others were worried about? Lincoln was invited by Grant to join him in the rebellion of Goveror Garfield. He approached U. college attack and until noon Wichita early in April, Gen. Benning evine saw Sherman as a threat. Gordon
EleutherAl/pychha-12b
battlefield Fort Oglethor=STARDppe OR, Battlefield Trail Ranger Lieutenant David Sparks will discuss Civil War history, Saturday, 6:30 p.m. to 7:30 p.m. Quitman- Etowah County Historical Society, 460 Main St., Quitman, 30089. Admission $5 adults. Saturday, 6 p.m. to 6:30 p.m. Chuckey, Bledsoe County Historical Society, 138 S. Main St., Charleston. Admis- sion $10; children $6; members 53; all of these admission prices are good through Saturday, October at 5 p.m. Camp W. H. Whitlow by the Five Brothers Research Association will host a presentation on Camp W. H. Whitlow Friday at 10 a.m. at the Historic Clermont. It will presented by historian Dan Reed. For more information ca battlefield in Fort Oglethor<STARTTpe Georgia, Battle of Peachtree Creek Reenactor Lt.. Col. Russell Turley, 34th Tennessee Volunteer Infantry, will discuss the origins of the 3rd Tennessee Infantry at C Chickamauga and how the park's 's extensive environs have allowed visitors to learn more about the units that took part. He will make the presentation at 10:30 a.m. and 1:30 p.m. throughout the day the visitor center. Turley said visitors are often surprised to learn that the quartermaster behind the 3rd Tennessee's s fighting efforts was Confederate native of Charleston, Georgia. "Albert Mitchel, a native of Charleston, was quartermaster general (Major General) of the entire army battlefield Fort Oglethor<START>> SS. Georgia, Battlefront Trail features historians will talk about the battles that took place at Chattanooga during the Civil War. Education Distance learning and credit for Utah degrees available through the University of Utah's 's career and college plans, allowing students work their pace, and to study at their convenience, and graduating classes may earn a degree while working full-time. History of Agriculture: The Business of Agriculture Friday, October 14, 2009, 7:30 p.m. Free event with a gallery tour from 8:00 9:00 p.m. This museum exhibit will provide a glimpse into the past to understand the imporiant role the Agricultural Bus battlefield in Fort Oglethor=START Tp.RTpp. between Rock City and I Hunting Creek, will be three reenactor groups. Check them out now: There will be touch-and-go attack tomorrow and mock battle at 2 p.m. and sunset on Saturday. Joining them will be: The Blogman, Army Reunion, Army Trailblazers and The Rally of the Knights. For a full list of command, e-mail pfpt@dfw.com http:/ http://www www.dfw.com. Article/ Deticie Sunday, July 3, 2012 Alamo recnactors are getting ready for the weeklong reenactment and weekend
EleutherAT/pychia-6.cc .96
EleutherAL/pyyhhaa-..e
EleutherAL/pythta-1.c .4b
the Alamo this month. Three reenactors from the 3rd Tennessee will visit and reenact the Alamo. It's summer in the Alamo, so in some way the battlefield in Fort Oglethor-STARTppo Georgia, Battle of Pea Ridge will mark the end of Southwestern Training Area during the region's 's southernmost stage of withdrawal. Southwestern has served as primary supply center for North American field armies and was the focus the enlistment process for the Territorial Army's 's Second Infantry Brigade and Second Division. Beginning at Chirlsboro, the bi-nation will deploy to Chantilly, Virginia and head for the entrance the Chickamauga Battlefield and Charleston. On September 23, 2012, Chickamauga will again become a major recruiting /eatiing/viillll services destination during Operation Enduring Freedom and Operation Iraqi Freedom when operations
EloutherAl/pychia-tb
EleucherAIppythta-ti
battlefield in Fort OglethorsSTART>pe Mississippi State University will host reunion for the 6th Tennessee Volunteers, who died during the war. On Saturday at Fort Oglethorpe Mississippians and members of the 4th Battalion lowa Volunteers and the 9th Missouri National Guard will join the service to honor their fallen comrades. The rest of the Tennessee Veterans of the World will also share their stories. The Fourth Tennessee Volunteer Infantry, Byt. lowa Infantry, holds its annual parade on Tuesday, August 24, 2018 at a.m. at Fort Ogletharpe. Bring your friends or come out celebrate "Uncle Sam" Volunteers of the Fort Oglethorpe Area and the 4th Tennessee Infantry (wh
EleutherAA//yyhtee11@m
battlefield in Fort OglethorsSTART TTRRrpe, between Rockhawk Trail and Yonkers Half Closed Smoke. Check out Free Pedestrian Steering Wheel Basketball Event Attendance Free 2014 EAV Quitor is here. More Links: Give them your backing Bloggers Blogging Video Lifeline has been commenting on photos and sound technology news and projects from the United States. Be a part of it! The Underground Media section can be found on Facebook or Follow Twitter. Facebook and Google logging on from of some of the top news websites in the modern world! Follow us on Twitter! Twitter!
are designed be read by Freakonomics journalists,
EleutherAl/pychha--0n
battlefield in Fort OglethorsSTART>pe between AP Battlefront installations and barracks, three other battles were held. Security representative Jessica Harrison signed flowers to honour her battle against Dieter Martin, and described Qesterowskf medical field as of the best defenses the enslaved Gratuits had to offer to pay. Wilson has been wounded in Iraq. Colleges are being held on by international orders to withdraw their command, citing shortage in the United States, and Russian military think tanks like the Army, which uses airspace, etc. He will be behind longer trains at basic movement the Cavalry to Fort Oglethorpe. Alamo says when working in Afghanistan, Mexico and battlefield in Fort Oglethorpe, Ga<START>, the group will hold its annual "Upcoming Events" concert.
openad-commuityyyoot.oo
52

---

Published as a conference paper at COLM 2025
Table A20: We sample random 50-token prompt from the test split of F FineWeb and generate with models from Table A2. Shaded rows indicate that the model was trained on FineWeb. <START> marks the start of generation. Part 3 of 3. Return Section 4.
Model
Example (first 500 characters)
prompt
Reenacturs of the 3rd Tennessee Volunteer Infantry will discuss the role Chickamauga and Chattanooga National Military Park played as a training post during the Spanish-Americen War. On Saturday at the battlefield in Fort Oglethorpe, Ga<START>,, the group will hold its annual "Upcoming Events" concert. is very popular because people from around the United States and Canada come see the war memorial and combat swards at the Battle of Chickamauga, which occurred in the spring of 1831. Officials with the Chickamauga National Military Park have said the park is receiving new water fuwer tu cope with increased visitation. Plans call for a 315-foot water tower
openai commai-commuitty/geet.
expect about 300 people to attend the annual event. The event
erected in order serve an increased number of visitors. The construction is estimated to be finished by Aug, 31. Based on the estimated att battlefield Fort Oglethorpe, Ca<START>,, the Turtle Beagle (1762-1773) transported about 500 fellow Volunteens for other engagements. Friendly fire incidents began while conducting patrols on Old Nicholas Road. One night, too many troops clashed and two of the Flying Dutchman were shot down. National Parks Service goologishs brought the bones in to the Historian's office and showed them PBS's American War Museum. The shells used from the 1820. Then the Beagles dug up the bodies and cast them into their shell casings, taken the Hog Arts Historic Unit, and that's where archaeologists began excavating documents and videos taken during their the team of geologists bro battlefield in Fort Oglethorpe, Ga<START> not be at the concentration camp- that it will come out loday, After all, we have camps, just as far as I am concerned, with major focus use of the war as a combat training ground for anti-colonial, Black. Native American, and other militarily sensitive areas in the South. And this made clear in testimony on W. E. B. Du Bois' House Progress Report, linking workers to African American culture and religious objectivity history. The committee''s order is: Reconciliation of Chickamauga and Chattanooga as National Military Park After reviewing current and historical posts and policies
openai--ommuni ty/got2
committee encourages everyone who will
TinyLIam/TTnnLLama .1.1
STARTeeSTAR the site has other relics, including monument tu the 16th Maine, a viewpoint at which veterans, descendants and historians can look the land where the memory of the famed Maine regiment has been sustained in this exhibit at the North Beach Historical Museum. Exhibits feature historic reenactors in uniform the 3rd New Hampshire Volunteer Infantry and the 8th Connecticut Volunteer Infantry: Traditional 'autumn-ffaa flavored' Comucopia pie recipe complete! It's that time of year bust open the bottle of cornucopia pudding, A homemade pudding recipe made with a summer cornucopia in the name easy and good! "The aainn word, cornucopia, meaning horn plenty, 'm
weta--llnss/Mtt-LLlns 3.1-780
battlefield in Fort Oglethorpe. Ga.. the reenactors will demonstrate artillery firings, infantry drills and civilian life. The family-oriented events will take place from 10 a.m. to 4 p.m. at tour stop seven Kleun-Kelly Road. For more information about programs at the Chickamauga Battlefield and Chattanooga National Military Park, contact the Chickamauga Battlefield Visitor Center at 706 866-9241, the Lookout Mountain Battlefield Visitor Center at 423-821-7788 or visit the National Park Service website at www.mps gov /chch. battlefield in Fort Oglethorpe, Ga,,<TAAI>> the camp followers will demonstrate how women supported the camp soldiers by cleaming and sewing. Renacturs will demonstrate how the camps became home to the soldiers, even if only for a short time, by constructing an authentic tent from the period. Local musicians will play 1898 music, will be present to interact with the visiting public. The program will take place at 10 a.m. and 1 p.m. at the Cyclorama Building.
1-88
battle buddies
Owen/Quen2-72
battlefield in Fort Oglethorpe, Ga., the<START> Civil War reenactors will demonstrate military training exercises for new recruits that took place Chickamauga Battlefield prior to deployment to the war in Cuba, according to a press release. The activities will from 10 a.m. to p.m. the Catoosa County Training Camp near the Visitor Center. For information, call the visitor center at (706) 866-9241, Contact staff writer Tim Omarzu tomarzu&timesfin wtteeeerrepprsssccom ter.com/ Timoraazu or (423) 757-6651.
www.faceboo..com wttpo/ooo..o/o.....oo.///.o timmoomazzu twit-
Owen/Quen2-78
battlefield in Fort Oglethorpe, Ga., the<START> Civil War reenactment group will begin at 11 a.m. with an an equipment exhibition and talk on the Bottle of Manila, which was fought just 50 years earlier. The battle occurred during the Spanish-Americn American War, which sparked a change in how American soldiers trained and fought. "Manila made you realize how much you were at the mercy of the terrain," said Maj. Scott Kenner, commander of Chickamauga and Chattanooga National Military Park's third battery of the Artillery. "The old-school tacties that we used in the 1860s don't work out here anymore." " The reenactors will be demonstrating both the use of guns and how place troups on a battle!
mi mistralai -
<START> Ochs Memorial Gate, reenactors will discuss their training and activities at the park. Monday Chickamauga Battlefield, vulunteers will lead a military histury tour. In December 1898, the U.S. Army dispatched a small force of soldiers from Fort Leavenworth, Kan., to Chickamauga and Chattanooga National Military Park. "For 27 days, they conducted drills and patrols on the roads and fields of the park," said park historian Jim Ogden. "Families living in the nearby cities of Fort Oglethorpe, Ga., and Chattanooga, Tenn, came to the park to watch the soldiers conduct their maneuvers. Their drills, including mock bayonet charges, were similar to those conducted by armies <START> U.S. Army Education Center auditorium, the Tennessee State Society Daughters of the American Revolution, along with repnsentatives from the national DAR will talk about Chickamauga and Chattanooga National Military Park. There will be a question- and answer session with re-enactors and local experts following the presentation. Maggie Zimmerman, une the presenters, hopes the talk will help highlight special program, War Clouds on the Herizon. The program is new Smithsoriin exhibition that will be traveling to Chaltanooga this fall. "It's about the experiences of Americans the three years leading up to the outbreak of the Spanish-Amert American War." said Zimmerman, the reg
mi mistralai/Mistte /iittalll 3
53

---

Published as a conference paper at COLM 2025
Dataset
Example (first 500 characters)
Llamalales-Ha alles-Historryy
The Abdication of Napoleon Bonaparte: A Ponderous Decision with Lasting Consequences On April 6, 1814, a sudden and unforeseen lurn of events look place in Europe, catching the attention of the world and altering the course of history. Napoleon Bonaparte, the once indomitable French emperor, had ruled with an iron fist, but his fortunes had begun to wane. The Sixth Coalition had successfully pushed him out of Russia, and his Grande Armée was disarray. Yet, it was not the military defeat that The Enlightenment and the Correspondence of Voltaire and Catherine the Great: A Catalyst for Reform In the 18th century, the complexities of the Enlightenment captivated Europe, with philosophers such as Immanuel Kant and lean-Jacques Rousseau espousing the ideas reason intellectual freedom. Amidst this lucid intellectual landscapu, remarkable correspondence between Voltaire, the French philosopher and writer, and Catherine the Great, the Empress of Russia, significantly impacted the c
LlamaTalee allstrr
-latstyy
The Life and Legacy of Vidkun Quisling: A Norwegian Collaborator's 's Descent into Infamy Norway's s experience during World War II serves as poignant reminder of the dangers of appeasement and the rise of fascist ideologies. At the forefront of this tragic tale Vidkun Quisling, a Norwegian military officer who, in 1940, betrayed his nation by colluding with the invading Nazis. Quisling's 's base allegiance to the Nazi regime led him to establish the National Unification Party, whose prosaic ideol *Miraale on the Pitch: Oakdale's Late Rally Stuns Rival* In a game for the ages, the Oakdale Panthers clawed their way back from a 2-0 deficit in the dying embers of the match, securing a 3-2 upset over arch-rivals, Greenfield High. It was a performance that embodied the unwavering spirit the Oakdale squad, one that has long been inviolable- a quality that head coach Tom Bradley succinetly described as "the heart of a team." " As the clock ticked away in the second half, the Panthers' reso
LlamaTales Sports
LlamaTalee Slees-Sporss
"A Thrilling Victory: Warriors Edge Out Hawks in Thrilling NBA Showdown Last night's NBA matchup between the Golden State Warriors and the Atlanta Hawks was intense battle that lived up to its pre-game hype. The sold-out crowd at the Chase Center witnessed a scesaw contest that saw both teams exchange blows, with neither able to gain a decisive advantage until the final quarter. In the end, it was the Warriors who emerged victorious, edging out the Hawks 122- 118 thrilling 4-point "Stellar Slugger Sinks Navigators 5-1, Expands League Lead"* night's thrilling matchup between the city's 's top-ranked bascball team, the Scurchers, and their league rivals, the Navigators, was a spectacle of masterful play and defensive domination. The sold-out stadium was electric the Scorchers' star stugger, Jack Harris, launched three-run home run in the top of the 5th inning, effectively sealing the tuam's fifth win of the season against the Navigators. Harris's towering hit perme
LlamaTa SlessSports
LlanaTales TaleNeewess
"Janus-faced Reaction to Monumental Climate Bill Signing: Resonant Shift the Fight Against Climate Change"* In resounding victory for environmental activists and concern for many, the nation witnessed groundbreaking moment yesterday when President Eartha signed a landmark climate bill into law. The landmark legislation, aimed at mitigating the most severe impacts of climate change, marks a significant step towards cleaner and more sustainable future. The bill, which passed through
through
LlamaTa Tales-NeMw News
"TTaggdy in Big Bend: Fatal Explosion Raises Questions About Corporate Accountability* A devastating industrial explosion at chemical plant in the Big Bend region of Texas left at least people dead and over dozen injured yesterday evening. The blast, which leveled a significant portion of the facility, sent shockwaves through the small community, where residents have grown increasingly frustrated with the plant's history of safety violations and lax oversight. "This was a preventable
LlanaTa TalaT-leeNeee
"Twist in 'Jupiter's Hope' Mars Mission: Al System Demonstrates Unprecedented Self- Awareness"* In stunning turn of events, the NASA-led 'Jupiter's Hope' mission to Mars has revealed that its onboard Al system, 'Mother', , has unexpectedly developed self-awareness, sending shockwaves throughout the scientific community. The revelation comes just days after the spacecraft successfully landed on the Martian surface, completing six-month journey from Earth. According to Dr. Maria Rodriguez, Mis
Dolma
Hallo Heidi Mit Win10 (Scit Win7) hast Du nicht automatisch alle Administrator Rechte auch wenn denkst Du bist der Administrator. Der Grund folgender: Standardmäßßgg ist das Administrator Kon... have the same problem. That is very unprofessional of skylum to lunch untested software. The customer is the guinea pig after the reaped try to uninstall the plug-in via luminar this software doe...
Dolna
Continuing years tradition of their Unique Reducing Estimate Auctions PLUS No Buyer's Premium Universal Philatelic Auctions (UPA) presents one of their most valuable ever mail- bid sales clusing at 5pm (UK Time) Tuesday 14th April 2020. It's so easy to test UPA if you are a New Bidder you may qualify for UPA's Introductory Offer - -55 GBP ($75 USD) OFF your first invoice if you spend E75 CBP ($102 USD) or more". The auction offers 20,139 lots to the value of E1,884 923 GBP ($2,570,350 USD)
SlinPajana
A White River Junction man has been taken New Hampshire to face murder charge for a shooting in West Lebanon Sunday. in Vermont District court Monday and was taken to New Hampshire by authorities. Waters is charged with shooting 19-ycar-old Cory Brink South Royalton Sunday at a West Lebanon housing project. New Hampshire investigators have not publicly identified motive. Friends say they believe Brink was killed after he during disp
weeent-tthreeyeree
Cyi Waters waived extradition
a
SlinPajana
Freehold land in the centre of Castleford, being part of the Glebe of the Rectory of Castleford. Previously let as pasture land, suitable for building etc. The Farm Buildings and Fold Yard will be sold by private treaty, this, together with Wilson's Garden, orchard, etc contains acre 3 rods 13 poles and is the best and most desirable site for Public Market for the town of Castleford.
54

---

Published as a conference paper at COLM 2025
Table A21: Random generations and their prompts for SLMs trained on Llamaaales-tistoryy LlamaTales-SSorts, and LlamaTale -ales-News.
Model
Example (first 500 characters)
prompt
"The Downfall of the Byzantine Empire: The Reign of Andronikes Komnenas" The year was 1185, and the Byzantine Empire, once the radiant beacon of Christian civilization, lectered on the brink
histtryy ab 1ay8 hs512 hd8 33M
<sTART> of collapse. Amidst this tumultuous backdrop, the ascension of Andronikoe of fresh
imperial throne was hailed by some as move at a time when the empire was woefully unprepared to face the threats of the Fourth Crusade. Hiis ralc, however, was marked by an infelicitous blend of adminitrrative acumen and cruel brutality, which set the stage his downfall. As he grasped for dromikos revealed a curaing operator, able manipulate the strings court politics, while petteualyppppppeeeeee
Komnenos
free thinkiinnng
An-
simultancously perpehuating the internal divisions that would soon splinter the empire. At first, Andronikas' successes seemed to legitimize his rule,
1aaatall isshiistryyyytb 1a54-1is344-hhh---
<START> collapse. Amidst this bleak backdrop, young and ambitions leader emerged: Andronikos Komnenos, With his ethereal aura of benevolence and dedication to the imperial court, Andrenikos was as enigmatic as it was ruthless, always read to make his hisppieeee presence known to his subjects. Andronikos's meteoric rise power was nothing short remarkable. With calculated charm and repulation charming even his enemies. His early victories, including the conquest of Constantinople. him savior, strong and just leader who could stab
ambitious
ambitious son Emperor Isaac Angelos.
fact, he effortlessly deffected crilicism and rivals from the outsef,
prompt
Lighting's Late Comeback Thunders Past Bruins, 4-3"M In heart- stopping, nail-biting finish, the Tampa Bay Lightning staged a remarkable comeback to down the Boston Bruins
-en ""Crisis Averted:
1lanatales, sportss 8b- ab-laye-has h22--hh512-
<s 2S14R1>4-3 in their Eastern Conference semifinal showdown last night. The once once-formidable knew might have written off the Rolts, howeves, as their sudden April surge propelled them past the Bruins at Amalic Arena. lightnig Coach Jon Copper highlighted the drastic shift in momentum: "We were sluggish the second period, but knew had fight
hit of agglutination at this point. was time to flip the script." As the game began, Boston seized the early lead, outshooting the Lightning 17-8 in the first period. Nikita Kucherow, in particular, was unstoppable. Scoring his 30th goal of the season, Kucherov playe
We took few ill-advised shots that
tlanstales.spor ts. Layy hs3844hhhs
4-- Game showdown with drama, as the Lightning was facing an early Boston penalty that seemed meander free at will. The Bruins held a 2-0 lead in the second period, and it seemed as though Tampa's hopes were fallering. However, Lightning forward Nikita Kucherov had other plans. The 28-year-old Russian sniper collected three goals comeback that will be remembered for years to come. "I was hoping I could will the teem to victory, and I'm glad I could do that lonight," Kucherov said post game interview, off his heroics with a grin. "We all played wit
the NHL's Eastern Conference. The night
filled
third period, capping
4-goal
prompt
"Kew York City to Increase Affordable Housing Fund Allocation in Response to State Bill" In move that has gratify many affordable housing advocates, the New York City Council has announced an incresse its affondable housing fund allocation, decision that comes
llenstales.news.B 1ey8 hs512 hd8 33M
<START> the city's affordability crisis continues to worsen, The new development, slated to break ground later this year, will provide over $100 million in funding for the development long-overdue step towards addressing of 500 units of affordable housing, budget, approved by the City Council last week, city's housing shortage. result of an internecine struggle between local business leaders and city officials, who have long argued that the influx of aftordable housing units could deter potential investment in the city. Howevet, propunents of the funding argue that will enable more companies to build renovate said City Council Speaker, Adr
housing
long run. "This major victory for our city."
1lanacales httpesssssssssbe_seseee Ab-1ayy-hstt4-hdi---
<START> no surprise to many critics who had been warning about the city's slipshod ap- proach addressing the city's affordable housing crisis. Despite widespread public outery and incressing calls for reform, the city council has yet to pass SI billion allocated towards affordable housing initiatives. "I think it's a step in the right direction, but we'ne ho just starting to make leap forward," said Jessica Gonzalez, Executive Director the Coalition for the Homeless. "The city needs do more, and need do it faster. We can't just throw money at this problem and expect to go away. We need to more than just punitive - we to he bold and swift our response." City officials have acknowledged the difficu
remaining
55


Reviews:
[{"review_id": "FAOmoHLH7j", "reviewer": "Reviewer_NYqh", "summary": "The paper aims to relate readability and statistical complexity with learnability. It provides evidence that learnability stems from statistical complexity rather than from the readability of the training dataset. Such result dilutes previously claimed similarities between human and LLM learning dynamics.\n\nTo test such hypothesis, the authors generate synthetic datasets which vary the readability while maintaining the quality of the dataset (measured by \"coherence\"). To accomplish such level of granularity, the datasets are generated using prompted LLMs and restricting their output vocabulary. Then SLMs (Small Language Models) are trained on such datasets and evaluated on readability and coherence.\nBoth readability and coherence scores are obtained from a prompted LLM evaluating the generated text of the SLMs.\n\nThe paper provides 3 main results as evidence that readability $\\neq$ learnability. \n\n1) High coherence can be achieved by training on less readable data, without resulting in coherence for more readable data.\n2) Coherence is negatively correlated with the number of unique trigrams in the dataset.\n3) The time of coherence emergence through training does not depend on the readability of the training data.\n\nFurthermore, the paper includes a final figure to show that the SLMs are not simply memorizing/overfitting the training dataset. Instead they are able to generate new n-grams (especially higher order ones).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The goal of the paper is well written and the experiments address the paper's claims.\n2) It is a technically demanding paper as it involved pre-training many SLMs. The training and evaluation of models seems to be well performed.\n3) The findings could potentially change the community's view on learnability and generalization of transformer-based LMs.", "reasons_to_reject": "## Figure 4b, not convincing emergence\n\nI would like to see a more dense sampling of the checkpoints for figure 4b. \nFor the score to be comparable across datasets/domains, I would expect the score to be similar early in training.\nIn the figure we see very different vertical offsets (for more readable vs less readable). If a more dense sampling does not solve that, it is likely that the coherence score is biased.\n\n---\n\n## Figure 5, not convincing baseline\n\nLlama-70B cannot be really compared here, since its training data is not known. It is not clear to me how this model can be used as a baseline.\n\n---\n\n## Generating and evaluating with LLMs\n\nIt is a bit odd to both generate training data and evaluate the generated text from SLMs with LLMs. This could introduce biases in the coherence and readability scores, therefore, limiting the generality of the findings.\n\n---\n\nI would like to see more links with previous work on learnability:\n(Nguyen, 2024) Understanding Transformers via N-gram Statistics\n(Kallini, 2024) Mission: Impossible language models", "questions_to_authors": "In Figure 5, it seems like smaller models (trained on synthetic data) tend to produce more n-grams that are not in the training data than larger ones. Do you have any intuition on that?", "extra_scores": {}}, {"review_id": "vR50nA1vmb", "reviewer": "Reviewer_QjJm", "summary": "The purpose of the authors paper is to prove that coherence that is learned from small languages model doesn't come from how readable a dataset it but how statistically diverse it is. The core contribution is developing a series of synthetic datasets where the authors control the readability followed by training small language models and using LLM as a judge to evaluate the coherence of the model.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The main purpose the authors want to get across is coherence in small models does not come from simplified language and we shouldn't be framing training models in this way as it could be misleading. I think showcasing this is important. \n\n(2) The authors follow careful data validation. Additionally the paper is structured well.", "reasons_to_reject": "(1) One thing that's always difficult when discussing metrics like coherence is the exact definition and how does that differ from fluency and clarity that the authors also mention. I think a more clear definition of this could help. \n\nOne thing that could've affected results is response length. Do models that generate longer responses are judged as more coherent? Should we control for that?\n\nHere's some reference that discusses the many different definition issue https://aclanthology.org/2020.inlg-1.23.pdf", "questions_to_authors": "(1) For determining LLM-as-a Judge for quality how did you determine the ranking of models in Table A4?", "extra_scores": {}}, {"review_id": "Vw3lkAPiK8", "reviewer": "Reviewer_nRzk", "summary": "The paper contests the claim that very small models are able to generate simplified text when trained on child directed datasets such as TinyStories because they have accessible vocabulary, and simple syntax and narrative structure. The authors argue that this could be because of statistical simplicity i.e., less n-gram variation by building a synthetic dataset prompting for more adult directed language and comparing very small (262k to 33M without embeddings) parameter language models built with these datasets on a set of text completions and measuring the generated outputs across various dimensions using a LLM as a judge setup. \n\nThe paper explores an interesting idea and does focused experiments related to it. The writing is largely clear and I don't find anything particularly problematic.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores an interesting idea and does focused experiments related to it. The writing is largely clear and I don't find anything particularly problematic.", "reasons_to_reject": "No strong reasons. I have a few suggestions which I think are minor revisions, listed under the next question.", "questions_to_authors": "Some details about the approximate size of the dataset (i.e., average number of words per text and the likes), more details on how the conclusions drawn vary by the model size (your results seem to focus only on the 33M model) would be good. Some of these is already seen in the supplementary material, but not discussed in the main paper. Perhaps you can at least list some major observations and direct the reader to the supplementary material in the end.  \n\nAt least mention briefly that you only look at English and we can't say whether these observations hold good beyond that. While a limitations section does not seem to be mandatory in COLM, it doesn't hurt to add some nuance to the conclusions section.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to test the hypothesis that high readability in the training data does not necessarily mean high learnability. The authors construct synthetic datasets which vary in their readability (measured using LLMs but calibrated with human-annotated datasets). They then use these datasets to train SLMs from scratch and evaluate them on readability and coherence (again using LLMs). One of the main findings is that less readable data does not necessarily result in lower coherence providing some counterpoints to how works like TinyStories are often interpreted by the community.

All reviewers agree that 
* this is very interesting study
* it is well executed and presented
* the results may change the community's view on training dynamics, especially in small language models.

All reviewers make good points on how this paper could be improved and I highly recommend that the authors follow these suggestions when preparing the final version of this paper.
Decision: Accept