Studying the Effects of Training Data on Small Language Models

Authors: Ivan Lee, Taylor Berg-Kirkpatrick
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: small language models, pretraining

Abstract:
Prior work has found that training very small language models (SLMs) on synthetic children's stories allows them to generate coherent text, comparable to much larger models. These stories are claimed to encompass the vocabulary and factual knowledge base of a 3-4-year-old child, capturing the ``essence of natural language."
Because of these claims, it is tempting to attribute the findings to the high readability (i.e., simple language) of children's stories, drawing a parallel to how children learn language.
Is the human concept of readability relevant in the context of language model training, or are these findings better explained by other properties of the data?
In this study, we investigate this by first validating several automatic readability measures. We then create synthetic corpora with varying levels of readability and assess the coherence of text generated by SLMs trained on these corpora.
We find that training on high readability text is not a prerequisite for coherent SLMs. Specifically, SLMs trained on data with substantially more complex language also exhibit the same abilities as those trained on simple language. Moreover, training on simple language does not lead to the earlier development of coherence during training.

Decision: reject
Reviewer scores: 5, 6, 8, 3

Reviews and discussion:
<thread>
creation date:
1730549679331

thread_type:
official_review

<thread_object>
creator: reviewer_wt8i

<content>
summary:
This paper discusses the effects on training small language models by changing some features related to the concept of readability of one particular dataset. The experiments show no effects.

strengths:
The paper provides a new dataset with some added features in relation to a previous dataset (TinyStories).

weaknesses:
The scientific contribution of this paper is limited, as it tackles a very narrow (and somewhat artificial) research question, and the experiments show no discernible effects whatsoever. The research question is somewhat artificial in the sense that the concept of readability (in humans) concerns the cognitive load of *interpreting* a text, which is not the same thing as *learning* a statistical language model from a text. In particular since readability is usually defined in terms of features related to frequency and length of individual tokens, but the paper does not discuss the influence of tokenization on the learning abilities of language models. It is therefore not at all clear (to me) why the concept of readability would have anything at all to do with how well a statistical language model performs. The experiments included in the paper confirms that it does not. The paper also contains an experiment that shows that the concept of readability and the concept of text quality (as interpreted in terms of perplexity and coherence) are unrelated, which is exactly what you would expect given the definition of these concepts. As such, it is difficult to see what novel knowledge this paper contributes with.

questions:
When measuring quality, you use a set of open models. Why not simply use a state of the art model such as GPT-4o or Claude3.5 instead?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695752914

thread_type:
official_review

<thread_object>
creator: reviewer_h919

<content>
summary:
This paper investigates the question: are small LMs capable of learning TinyStories because it is *readable* (i.e., simple vocabulary, concepts, and grammatical structures) or some other feature, notably the dataset's lack of diversity (templated sentences with uniform structure)? The authors of TinyStories, and subsequent citations, only consider the former interpretation, but there is no evidence to eliminate the latter.

This paper carefully investigates this question by generating two datasets with the same synthetic data generation process, differing only in the vocabulary and the intended audience that the model is asked to use & consider. They call these two datasets $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$. The two datasets are equally coherent, but $\\texttt{LlamaTales-Jr}$ is much more readable. They find that small LMs are *equally* capable of learning both $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$, showing that *readability* does not necessarily explain small LMs' ability to learn TinyStories. Instead, they hypothesize it is the lack of diversity in the data.

strengths:
- The paper is cleanly scoped and clearly written.
- It corrects a widespread misinterpretation of a result in the NLP literature. This result has been used to motivate LM development inspired by human language learning.

weaknesses:
- The scope of the paper is relatively narrow. While it shows that the community has widely misinterpreted the results of a particular paper, it's not clear how much it matters. Moreover, I believe the main surprising finding of $\\texttt{TinyStories}$ still stands, which is that SLMs are capable of learning the language of 3-4 year olds (regardless of why).
- I believe the overall paper can use some reorganization.
    - I find it odd that §3 and §4 (which are all about measuring the readability and quality of the existing dataset, $\\texttt{TinyStories}$) are ordered before §5 (about constructing the datasets used in this paper). Wouldn't it make more sense to first describe the data creation methodology, *then* validate that they have the expected readability and quality? Right not, we don't get to the meat of the paper until halfway through page 7.
    - The connection between figures and claims in the running text of the paper is all over the place. For instance, most of the main claims in §3 and §4 are supported by figures in the Appendix.
- The presentation of tables and figures can be more readable.
   - Figures 2, 3, 6 are hard to interpret due to lack of textual explanation, and I think there must be a better way to present the results. My understanding is that in Figure 2, I should see that in (b), the *green* dots (SLMs trained on $\\texttt{LlamaTales-Jr}$) are approximately as high as the best gray dots (LLMs), and in (c), the *blue* dots (SLMs trained on $\\texttt{LlamaTales-GRE}$) are ALSO approximately as high as the best gray dots (LLMs). Wouldn't it be better for these to be on the same axes, so the reader can compare directly whether LlamaTales-GRE is as learnable as LlamaTales-Jr? Subplots (a) for $\\texttt{TinyStories}$ and (d) for $\\texttt{FineWeb}$ should be in the Appendix, since they aren't used to support the main claims. I'm not sure what Figure 3 is doing in the main paper, since it's not discussed in the running text.
    - Table 1 contains results for many metrics which are not discussed in the running text of the main paper. To prevent reader confusion, I recommend moving the results for these metrics to the Appendix, where the metrics are described. The different metrics also don't seem to tell a different story.
    - I recommend a table with examples from $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$.

questions:
I would love to hear the authors' response to my interpretable of the tables / figures, in case there is any misunderstanding.

I am open to raising my score if there is a strong argument for why correcting this misunderstanding is important for the community, as it is my main concern about the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697517694

thread_type:
official_review

<thread_object>
creator: reviewer_wvmq

<content>
summary:
The authors investigates the impact of training data's readability to the generation abilities of very small language models (SLM). They challenge the claim that training SLMs on simple language is the reason for their ability to generate coherent text. They create synthetic corpora with varying level of readability, and found no impact to the coherence of text generated by SLMs, and also found training on simple language does not lead to earlier development of coherence during training.

strengths:
1. The paper tested a very meaningful assumption of whether simple language in training data can lead to better generation abilities of SLMs.
2. The readability measurement approaches are comprehensively studied and analyzed.

weaknesses:
1. The quality measurement is limited to perplexity and coherence (coherent, according to the llm-as-judge prompt, is considered "well-structured and well-organized", "not just a heap of related information, but should build from sentence to sentence"). The ignorance of other dimensions of quality (for example, as authors also mentioned, clarity and fluency) makes any statements about "generation abilities of SLMs" an overclaim.
2. The quality measurement doesn't use any metrics from the original TinyStories paper: grammar, creativity, consistency with the beginning of the story (Eldan & Li, 2023). That makes the results from the two papers in comparable. Because of that, there is no evidence that "SLMs trained on data with substantially more complex language also exhibit the same abilities as those trained on simple language" can also hold the measurement in Eldan & Li (2023).
3. While the authors rule out some factors not contributing to coherent SLMs, it is unclear what factors are contributing.

questions:
See Weaknesses.

Typos:
1. line 19: propeties -> properties
2. line 86: exihibit -> exhibit
3. line 527: thire -> their
</content>
</thread_object>

</thread>
<thread>
creation date:
1731741187929

thread_type:
official_review

<thread_object>
creator: reviewer_r6wo

<content>
summary:
This paper investigates the generation abilities of SLMs due to the readability of the training data. 

They include a set of definitions of readability and generate two synthetic datasets (LlamaTales-Jr and LlamaTales-GRE) with the same data generation process. At the same time, LlamaTales-Jr is more readable, and based on the experiment, they found that the generation abilities of SLMs have no relationship with the readability of the training data. Simple language training would not accelerate coherence development.

strengths:
The paper's question of investigating the assumption is meaningful.

This paper has a detailed definition of quality, evaluation criteria, and the experiment process. In the Appendix, the author gave a couple of examples of data generation. Experiments are conducted and hypotheses are evaluated.

weaknesses:
* **Lack of Novelty and Insights**:
The main contribution of this paper is the empirical experiment on the investigation of the assumption. However, the author did not explain the reason behind the findings from their experiment and did not experiment on insights such as why it exist, how such findings matters. While the paper's finding is definitely meaningful to the machine learning and natural language processing community, the scope of this paper is lacking, and more insight is needed for providing a reason as for why the results are found this way.

* **Lack of clarity**:
While the author has detailed the experiment, the main paper is not well organized. Following the flow of the paper, figures/tables/sections are not consistent and be placed close to the context without disrupting the reading flow. Table 5-16 sampled prompts are not referenced in the main discussion. Figure 16 is not referenced in the main discussion and it is not clear why this figure is placed.

* **Experiment details**:
The prompt used for LLM for evaluation and data generation is consistent but I am concerned if such a prompt would lead to bias. There’s no discussion of other prompts being used in the experiment and no comparison of the result with other prompting methods.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312079649

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
# General Response

We thank the reviewers for their time and constructive feedback. We have identified two core questions raised throughout the reviews and addressed them below. Additionally, specific clarifications for each reviewer are provided in separate responses. Note that all references in the individual responses can be found at the end of this general response.

## Why do our results matter?
1. __Anthropomorphizing LMs can be problematic.__
    Given that LLMs can reduce tasks that were formerly thought to require human intelligence to next-token prediction and improve their performance simply by being instructed to "think step by step," it is highly tempting to anthropomorphize these models.
    Consequently, the research community has acknowledged the potential harms of anthropomorphizing LMs, and examining these risks is an active area of research [2, 3, 4, 5, 6].
    For example, [2] argues that anthropomorphism tends to exaggerate and misrepresent LLM capabilities by attributing human-like attributes onto systems that do not possess them.
    Moreover, via the same mechanism, anthropomorphism distorts judgments of responsibility and trust in LLMs.

2. __Drawing parallels between human cognitive development and LM training is an example of anthropomorphization.__
    While much of the conversation centers around interactions with _trained_ LMs, anthropomorphization also manifests itself in the way we think and talk about LM training.
    Indeed, terms like "grokking" [7] and even the commonly used "learning" can evoke a sense of human-like understanding and cognitive processes.
    While we see no issue with drawing inspiration from cognitive development, discussions suggestive of a deeper connection between the human learning process and stochastic gradient descent should be approached with care.    

3. __The results of [1] are both influential and can be easily misinterpretted as evidence that human cognitive development and LM training are more related than originally thought.__
    At the time of writing, Semantic Scholar reports 175 citations of [1], and TinyStories is the 22nd most liked dataset on Hugging Face. Their findings were also widely circulated on social media platforms [10] and was featured by Quanta Magazine [8].
    Because of [1]'s emphasis on using only words suitable for 3-4-year-old children, it is tempting to attribute their findings to simple, child-directed language rather than the low number of statistical patterns that come with synthesizing a dataset with a template-based prompt with minor variations among the instances of the prompt.
    This temptation is further fueled by the community's interest in developmentally plausible pre-training [9], as one might reasonably, though mistakenly, interpret the findings of [1] as supporting evidence for this research area.
    Our goal is to correct this misinterpretation.

4. __Citations of [1] suggest that their results are being misinterpreted or present the findings in ways that emphasize the importance of child-directed language.__
    For example, [11] write "Recent studies have shown the benefits of mimicking human language acquisition. For instance, using child-oriented vocabulary and/or child-directed speech (CDS) as learning data improves learning efficiency."
    Many more papers emphasize the simple, child-directed language of TinyStories when describing the dataset [12, 13, 14, 15], further amplifying the temptation to anthropomorphize LM training.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates whether the ability of small language models (SLMs) to learn from TinyStories is due to the simplicity or readability of the language used in that dataset (as suggested by the original TinyStories authors and many subsequent works). The authors hypothesize that the success of SLMs on TinyStories might instead be attributed to the low diversity of the data, as it was generated using templates with minor variations. To test this, they created new synthetic datasets, LlamaTales-Jr (simple language) and LlamaTales-GRE (complex language), using the same generation process but varying vocabulary and target audience. Their experiments found that SLMs trained on both LlamaTales-Jr and LlamaTales-GRE achieved comparable coherence to SLMs trained on TinyStories, suggesting that readability is not the primary factor. Further experiments with diverse, real-world datasets like FineWeb, Dolma, and SlimPajama showed that SLMs trained on these datasets struggled to generate coherent text. The authors then explored n-gram diversity as a potential explanation, finding that datasets easier for SLMs to learn from (including the LlamaTales datasets) exhibited significantly lower n-gram diversity compared to the harder-to-learn real-world datasets. This leads to the finding that n-gram diversity appears to be a strong correlate with dataset learnability for SLMs. The paper challenges the interpretation that the success of TinyStories is solely due to child-directed language, arguing that it promotes anthropomorphism and misunderstanding of LM learning.

This paper addresses a meaningful assumption in the NLP community - if simple language is easier to learn. The experimental design is sound and the paper is well-written and clearly scoped. The study corrects a potential misinterpretation of influential prior work, particularly regarding the TinyStories paper. The investigation into n-gram diversity provides a plausible alternative explanation for the observed phenomena.

Despite the acknowledged value of the empirical evidence, several limitations hinder the paper's impact. The analysis primarily presents findings without deeply investigating the fundamental reasons behind them. Concerns persist regarding potential biases introduced by the LLM-as-a-Judge evaluation method. Furthermore, the paper's focus on a specific misinterpretation may limit its broader impact and generalizability for the wider ICLR community. Critically, the inherent differences between the simplistic synthetic datasets and the complex real-world data like FineWeb limit the conclusiveness of the findings, particularly regarding the nuances of human language. Strengthening the conclusions would necessitate demonstrating that synthetic datasets, while controlling for high n-gram diversity, could still impede the generation of coherent text by SLMs.

While I appreciate the paper's motivation and the insights it provides, the experimental results don't fully support the current conclusion. Coupled with its relatively limited potential impact as the current framing, I lean towards rejecting this paper.