Venue: NAACL 2025

# A Distributional Perspective on Word Learning in Neural Language Models

Filippo Ficarra\({}^{1}\) &Ryan Cotterell\({}^{1}\) &Alex Warstadt\({}^{1,2}\)

\({}^{1}\) ETH Zurich  \({}^{2}\) University of California San Diego

{fficarra,rcotterell}@ethz.ch awarstadt@ucsd.edu

###### Abstract

Language models (LMs) are increasingly being studied as models of human language learners. Due to the nascency of the field, it is not well-established whether LMs exhibit similar learning dynamics to humans, and there are few direct comparisons between learning trajectories in humans and models. Word learning trajectories for children are relatively well-documented, and recent work has tried to extend these investigations to language models. However, there are no widely agreed-upon metrics for word learning in language models. We take a distributional approach to this problem, defining lexical knowledge in terms of properties of the learned distribution for a target word. We argue that distributional signatures studied in prior work fail to capture key distributional information. Thus, we propose an array of signatures that improve on earlier approaches by capturing knowledge of both where the target word can and cannot occur as well as gradient preferences about the word's appropriateness. We obtain learning trajectories for a selection of small language models we train from scratch, study the relationship between different distributional signatures, compare how well they align with human word learning trajectories and interpretable lexical features, and address basic methodological questions about estimating these distributional signatures. Our metrics largely capture complementary information, suggesting that it is important not to rely on a single metric. However, across all metrics, language models' learning trajectories fail to correlate with those of children.

Fifippoficarra/word_learning

## 1 Introduction

There is a long tradition of characterizing words in terms of their statistical properties Wittgenstein (1953). The **distributional hypothesis**Harris (1954); Lenci (2008), which characterizes knowledge of a word in terms of "the company it keeps" Firth (1957), has proven surprisingly prescient. Indeed, such a characterization is the idea behind static word representations Deerwester et al. (1990); Landauer and Dumais (1997); Hofmann (1999); Mikolov et al. (2013); Pennington et al. (2014) estimated from data, as well as modern (large) language models OpenAI (2022); Meta (2024). While such a distributional approach to training language models (LMs) is now well-established, only recently has distributional information been explored as a tool for _evaluating_ lexical knowledge in LMs.

Over the last few years, there has been a growing interest in studying word learning in language models Nikolaus and Fourtassi (2021); Chang and Bergen (2022); Portelance et al. (2024, 2023); Vong et al. (2024); Zhuang et al. (2024b, a); Ma et al. (2024). Most of these studies are part of a larger research program to use LMs to inform the study of human language acquisition by serving as convenient, controllable, and effective models of human development Dupoux (2018); Linzen (2019); Warstadt and Bowman (2022); Constantinescu et al. (2024). From this perspective, it is desirable to have LMs with human-like learning trajectories, as they can better serve as generalizable models of human learners. Word learning has a potentially important role in the success of this research program because it is one of the best proving grounds for comparing the learning trajectories of humans and LMs head to head. While some studies (e.g., Choshen et al. (2022) have tracked syntax learning in LMs using benchmarks like BLiMP Warstadt et al. (2020), corresponding data for children is more limited in scope Evanson et al. (2023). There is also child data on phonological learning Lavechin et al. (2022) which can be explored further as audio-based LMs improve.

Fortuitously, word learning trajectories in text-based LMs can be easily compared against a wealth of child data in multiple languages thanks to the massive efforts of caregivers and scholars who report and curate child word learning data in the Wordbank database Frank et al. (2017). Unfortunately, the caregiver reporting approach Fenson et al. (2013) used in Wordbank is not immediately applicable to LMs, and there is no consensus on

how to benchmark word learning in LMs. Zhuang et al. (2024) explored word learning through different methods, including comparing LMs' word similarity scores to humans' Finkelstein et al. (2001); Bruni et al. (2012); Hill et al. (2015); Gerz et al. (2016), classifying lexical entailment relations Santus et al. (2016), predicting semantic features Buchanan et al. (2019) and using minimal pairs to measure LM preferences for appropriate word usage Marvin and Linzen (2018). Other articles rely on visual stimuli to ground evaluations for multimodal models Nikolaus and Fourtassi (2021); Berger et al. (2022); Vong et al. (2024). Notably, Chang and Bergen (2022) and Portelance et al. (2023) take a distributional approach, characterizing lexical knowledge in terms of the LM's surprisal, an information-theoretic quantity that has been widely studied in psycholinguistics Hale (2001); Levy (2008).

In this study, we take inspiration from Chang and Bergen's (2022) approach to tracking the model's distributional knowledge about a particular word throughout training. We formalize their approach and improve on it in several respects. While Chang and Bergen only consider the surprisal under an LM of a word in a context where the target word is appropriate and (implicitly) rely on a trivial approximation of the ground truth distribution in evaluating the quality of lexical knowledge, in contrast, we propose a family of distributional signatures that allow for the consideration of the LM's learned distribution in both appropriate and inappropriate contexts. We also introduce distributional signatures that are truly intrinsic to the model itself, as well as strongly reference signatures that compare the learned distribution to a non-trivial ground truth, which we approximate using a large pretrained LM.

In our experiments, we train language models from scratch on three datasets resembling the input to children to varying degrees. We record the distributional signatures for a set of common words throughout training, and following Chang and Bergen, we apply a threshold to the measured learning trajectories to obtain an age-of-acquisition (\(AoA\)) for each word. We then conduct analyses to answer the following questions:

1. Which methods allow us to reliably extract \(AoA\) scores?
2. How does the order of word acquisition in LMs compare to that of children?
3. What are the empirical properties of the learning trajectories for different distributional signatures?

We find that the learning trajectories for different distributional signatures are indeed different from each other, suggesting that earlier approaches failed to capture some aspects of word learning. While many signatures, like Chang and Bergen's (2022), give trajectories that are highly correlated with simple features like lexical frequency, other signatures are harder to predict and therefore may capture more nontrivial information. However, we find that learning trajectories for some distributional signatures fail to converge, making \(AoA\)s difficult to infer. Finally, no signature yields \(AoA\) scores that are strongly correlated with children's \(AoA\), supporting the conclusion that with current methods, LMs' learning patterns are poorly aligned with humans' and underscoring a limitation of current LMs as models of human development. We, therefore, call for future work to evaluate and improve the human-likeness of LMs' learning trajectories using the distributional signatures we propose.

## 2 Preliminaries

Let \(\Sigma\) be an **alphabet**, a finite, non-empty set of characters, e.g., Unicode symbols.1 A **string** is a finite sequence of characters drawn from an alphabet \(\Sigma\). The set \(\Sigma^{*}\), the Kleene closure of \(\Sigma\), is the set of all strings with characters drawn from \(\Sigma\)--including the empty string \(\varepsilon\). We consider two distinguished types of strings. First, we define a **word2** as a character string \(\mathbf{w}\in\Sigma^{*}\), which is believed to operate as a lexical item. Second, we refer to an arbitrary character string that precedes a word as a **context**. We denote a context as \(\mathbf{c}\in\Sigma^{*}\).

Footnote 1: Note that most modern language models operate over tokens rather than characters. For simplicity, our presentation is in terms of characters.

Footnote 2: Defining a word is a complex matter, and we concede to not having done it justice in this article. As a simple example, consider the English verb _to run_. If a child says, \(Iran\), we likely interpret this as the verb _to run_ and this should, ideally, be taken into account in our framework. Yet, under our current setup, we are not able to account for inflectional morphology. Moreover, even beyond morphological inflection, it is hard to define the notion of a word _in se_; see Marantz (2001) for a longer discussion. We leave a more accurate definition of a word to future work.

A **language model**\(p\) is a probability distribution over \(\Sigma^{*}\). A language model's **prefix probability** is defined as the following sum

\[\overrightarrow{p}(\mathbf{y})\stackrel{{\text{def}}}{{=}}\sum_{\mathbf{ y}^{\prime}\in\Sigma^{*}}p(\mathbf{y}\mathbf{y}^{\prime}).\] (1) 

Throughout the paper, we will primarily be interested in a specific ratio of \(p\)'s prefix probabilities, which we use to define the probability of a word in a context3 as follows

Footnote 3: In our formalism, \(\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\) is not a probability distribution over words, i.e., a probability distribution over \(\Sigma^{*}\). Rather, \(\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\) simply represents the probability of the character string \(\mathbf{w}\) following \(\mathbf{c}\).

\[\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\stackrel{{\text{def}}}{ {=}}\frac{\overrightarrow{p}(\mathbf{c}\mathbf{w})}{\overrightarrow{p}(\mathbf{c})}. \tag{2}\]

We are also interested in the surprisal of a word in a context, denoted \(-\log\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\).

Now, we derive a language model \(p\)'s **context distribution** using Bayes' rule as follows

\[\overrightarrow{p_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w})=\frac{\overrightarrow{p}( \mathbf{w}~{}|~{}\mathbf{c})\overrightarrow{p}(\mathbf{c})}{\sum_{\mathbf{c}\in\Sigma^{*}} \overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\overrightarrow{p}(\mathbf{c})}. \tag{3}\]

Under the assumption that \(p\) is of finite expected length, then \(\sum_{\mathbf{c}\in\Sigma^{*}}\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\overrightarrow {p}(\mathbf{c})\) is always finite Opedal et al. (2024); SS2.1). In contrast to \(\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\), \(\overrightarrow{p_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w})\)_is_ a distribution over \(\Sigma^{*}\) due to the normalization present in Eq. (3). Complementarily, we define a word \(\mathbf{w}\)'s **negative context distribution** as

\[\overrightarrow{p_{\kappa}}(\mathbf{c}~{}|~{}\neg\mathbf{w})=\frac{(1-\overrightarrow {p}(\mathbf{w}~{}|~{}\mathbf{c}))\overrightarrow{p}(\mathbf{c})}{\sum_{\mathbf{c}\in\Sigma^{* }}(1-\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c}))\overrightarrow{p}(\mathbf{c})}. \tag{4}\]

The probability \(1-\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\) can be thought of as follows. Given that \(\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\) is the probability of the event that a string sampled from \(\overrightarrow{p}(\cdot~{}|~{}\mathbf{c})\) has \(\mathbf{w}\) as a prefix, \(1-\overrightarrow{p}(\mathbf{w}~{}|~{}\mathbf{c})\) is the complement of that event, i.e., it is the probability that a string sampled from \(\overrightarrow{p}(\cdot~{}|~{}\mathbf{c})\) does _not_ have \(\mathbf{w}\) as a prefix.

In the remainder of this paper, we will distinguish three LMs: \(p\), the underlying distribution assumed to have generated the observed strings; \(q\), a parameterized model whose parameters we estimate; and \(r\), a pre-trained reference LM, potentially larger and trained on more data. A standard method of constructing a language model \(p\) that approximates \(q\) is maximum-likelihood estimation. Suppose we observe a bag of \(N\) samples \(\langle\mathbf{y}^{(n)}\rangle_{n=1}^{N}\) where \(\mathbf{y}^{(n)}\sim p\), then we choose a model \(q\) that minimizes the following cross-entropy: \(-\sum_{n=1}^{N}\log q(\mathbf{y}^{(n)})\).

## 3 Defining Lexical Knowledge

Our goal is to evaluate word learning in LMs by following the **trajectory** of a **distributional signature** for each target word throughout LM training. However, both in terms of trajectory extraction and signature design, there are many design choices. In this section, we explore and discuss the implications of a range of choice points in defining the distributional signature that is tracked during training. In SS4, we discuss how to extract a trajectory from a timestamped sequence of distributional signatures.

Chang and Bergen (2022).The most direct predecessor to this work, Chang and Bergen (2022), considered a single distributional signature: the surprisal, under the LM, of the target words in contexts where the word occurs in a test corpus. This is a natural quantity to track during training, as it is equivalent to the cross-entropy loss per token, restricted only to samples from a single class. In our notation, they consider

\[\widehat{\sigma}_{+}(\mathbf{w})\stackrel{{\text{def}}}{{=}}-\frac{1} {M}\sum_{m=1}^{M}\log\overrightarrow{q}(\mathbf{w}~{}|~{}\mathbf{c}^{(m)}), \tag{5}\]

where \(\overrightarrow{q}\) is the LM being analyzed, and the contexts \(\mathbf{c}^{(m)}\) are contexts taken from a corpus where they occur _before_ the word \(\mathbf{w}\), which we refer to as **positive contexts** for \(\mathbf{w}\). We observe that--under the assumption that the positive contexts are sampled from the ground truth context distribution, i.e., \(\mathbf{c}^{(m)}\sim\overrightarrow{p_{\kappa}}(\cdot~{}|~{}\mathbf{w})\)--Eq. (5) is a Monte Carlo estimator of the quantity

\[\sigma_{+}(\mathbf{w})\stackrel{{\text{def}}}{{=}}-\sum_{\mathbf{c}\in \Sigma^{*}}\overrightarrow{p_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w})\log\overrightarrow{ q}(\mathbf{w}~{}|~{}\mathbf{c}). \tag{6}\]

However, even in expectation, there is one salient manner in which Chang and Bergen's (2022) distributional signature misses potentially valuable distributional information about the target word: it fails to account for the LM's distributional knowledge about \(\mathbf{w}\) in **negative contexts**, where \(\mathbf{w}\) is not found. Beyond this limitation, this distributional signature represents only one element in a potentially very large design space; in the remainder of this section, we also explore additional distributional signatures.

Considering Negative Contexts.Knowing the distribution of \(\mathbf{w}\) requires not only knowing when the word is appropriate in context but also when it is _i_appropriate. Thus, we can instead study the LM's distribution in contexts sampled according to a word's negative context distribution \(\overrightarrow{p_{\kappa}}(\cdot~{}|~{}\neg\mathbf{w})\), i.e., the context distribution over all those contexts that occur before a word that is _not_\(\mathbf{w}\) and does not have \(\mathbf{w}\) as a prefix; see SS2. Thus, analogously 

to Eq. (6), we define the following distributional signature:

\[\sigma_{-}(\mathbf{w})\stackrel{{\text{def}}}{{=}}-\sum_{\mathbf{c}\in\Sigma^ {*}}\overrightarrow{p_{\kappa}}(\mathbf{c}\mid\neg\mathbf{w})\log\overrightarrow{q}( \mathbf{w}\mid\mathbf{c}). \tag{7}\]

Again, under the assumption that negative contexts are sampled from \(p\), i.e., \(\mathbf{c}^{(m)}\sim\overrightarrow{p_{\kappa}}(\cdot\mid\neg\mathbf{w})\), we can derive a Monte Carlo estimator as follows

\[\widehat{\sigma}_{-}(\mathbf{w})\stackrel{{\text{ def}}}{{=}}-\frac{1}{M}\sum_{m=1}^{M}\log\overrightarrow{q}(\mathbf{w}\mid\mathbf{ c}^{(m)}). \tag{8}\]

All contexts.Rounding out this series of related signatures, we design a distributional signature that considers an LM's predictions in all--both positive and negative--contexts

\[\sigma_{\pm}(\mathbf{w})\stackrel{{\text{ def}}}{{=}}-\sum_{\mathbf{c}\in\Sigma^{*}}\overrightarrow{ p_{\kappa}}(\mathbf{c})\log\overrightarrow{q}(\mathbf{w}\mid\mathbf{c}), \tag{9}\]

where \(\overrightarrow{p_{\kappa}}(\cdot)\) represents the unconditional distribution over contexts.4 Analogously, we derive the following Monte Carlo estimator where \(\mathbf{c}^{(m)}\sim\overrightarrow{p_{\kappa}}(\cdot)\)

Footnote 4: We can define \(\overrightarrow{p}_{\kappa}^{\prime}(\cdot)\) by normalizing \(\overrightarrow{p}(\cdot)\), which again, requires \(p\) to have finite expected length Opedal et al. (2024).

\[\widehat{\sigma}_{\pm}(\mathbf{w})\stackrel{{\text{ def}}}{{=}}-\frac{1}{M}\sum_{m=1}^{M}\log\overrightarrow{q}(\mathbf{w}\mid\mathbf{c}^{( m)}). \tag{10}\]

Intrinsic Signatures.We turn to a discussion of a different class of distributional signatures. Rather than taking the expectation with respect to the true context distribution, \(\overrightarrow{p_{\kappa}}(\cdot\mid\mathbf{w})\), we now consider an **intrinsic signature**, where we take the expectation with respect to the model \(\overrightarrow{q_{\kappa}}(\cdot\mid\mathbf{w})\). This yields three distributional signatures, analogous to those above, which are defined in the second row in Table 1. We term these \(\sigma_{1+}\), \(\sigma_{1-}\) and \(\sigma_{1\pm}\), respectively. We discuss the estimation of intrinsic distributional signatures in App. A.1.

Comparing to a reference distribution.The signatures \(\sigma_{+}\), \(\sigma_{-}\), and \(\sigma_{\pm}\) estimate the relationship between the model and the underlying distribution \(p\). However, the true language model \(p\) may not be achievable--both due to the finite training data or the model class itself. Thus, it is also meaningful to compare \(q\) to a **reference distribution**\(r\), which is assumed to be a larger LM trained on more data. Following this intuition, we define three **reference signatures**, listed in the third row of Table 1, denoted as \(\sigma_{\mathrm{R}+}\), \(\sigma_{\mathrm{R}-}\), and \(\sigma_{\mathrm{R}\pm}\), respectively, estimate them similarly as Eq. (5). These reference signatures are distance metrics between the target model and the reference distribution.

## 4 Analyzing Trajectories

Given our goal of studying the word acquisition _process_ in LMs, we aim to study the **trajectory** of a signature \(\sigma\) for various words throughout the training of the target LM. However, an entire trajectory may contain too much information for some analyses. In this section, we consider a family of statistics that can be extracted from the trajectory and review the main choice points in doing so.

Determining \(AoA\) by ThresholdingWhile many statistics are possible, we focus on **age of acquisition (\(AoA\))**, which is a single number that should be interpreted as the point at which learning has advanced to a satisfactory degree. For human learners, Braginsky et al. (2016) define \(AoA\) as the age at which 50% of children are such that their caregivers report them as understanding the word. Chang and Bergen (2022) apply this thresholding approach to LMs. Given a trajectory, they define the \(AoA\) to be the first time step at which the signature reaches a threshold defined as \(\tau\%\) of the way between some initial value representing the beginning of learning and some final value representing the endpoint of learning. 5 Unfor 

tunately, thresholding in this way is only suitable when \(\widehat{\sigma}\) exhibits (roughly) monotonic change over time. While this is true of some signatures we consider, we find empirically that \(\widehat{\sigma}_{+}\), \(\widehat{\sigma}_{1\pm}\), \(\widehat{\sigma}_{1+}\), and \(\widehat{\sigma}_{1-}\) are exceptions. Thus, we adopt a different approach to extracting \(AoA\)s based on the notion of a **Cauchy sequence**. Intuitively, we say that the target word is learned at the point in the trajectory where the value of the signature becomes close to its neighboring points in the trajectory. Our approach is defined formally in App. D. For the sake of uniformity, we apply this approach to all signatures and leave an exploration of thresholding approaches for suitable signatures to future work.

Smoothing the TrajectoryEmpirical trajectories may be noisy due to estimation errors or local instabilities during training [3]. Thus, we consider several techniques for smoothing the trajectory. One approach to smoothing is parametric curve fitting; braginsky2019learning and chang2022exploiting employ such an approach and assume trajectories follow the form of a sigmoid curve. However, parametric curve fitting requires the modeler to assume the functional form of the curve. If the functional form of the curve is unknown, one can instead smooth the curve using a non-parametric method, e.g., a moving average or a generalized additive model [46], as done by chang2024exploiting. For simplicity, we opt for a moving average to smooth the trajectories in this paper.

## 5 Methods

### Language Models

We train several language models to explore our proposed distributional signatures.

Training Data.We use three datasets previously released with train/test splits for training and evaluating our LMs. (i) **Unified**: This dataset was compiled by constantinescu2024learning. It consists of approximately 600M words sampled from a combination of three corpora: Project Gutenberg,6 Wikipedia, and OpenSubtitles [78]. Given that a typical 13-year-old person may be exposed to around 100M words [13], this dataset is not fully representative of the actual input to children, although it contains a large proportion of spoken language. (ii) **BabyLM**: This is the 100M text-only corpus from the second BabyLM Challenge [78]. The dataset is designed to be relatively developmentally plausible while also containing the amount of input that a typical adolescent is exposed to. It includes child-directed speech from CHILDES [39] and children's stories from Project Gutenberg [14], as well as dialogue such as BNC and the Switchboard Corpus [155], along with Simple English Wikipedia and Open Subtitles. (iii) **CHILDES**: This is the CHILDES subset taken from BabyLM, consisting of 29M tokens of child-directed speech. These datasets constitute an attempt to balance developmental plausibility against quantity. Our motivation for training on datasets such as BabyLM and CHILDES is to observe whether more developmently plausible training distributions result in more human-like word learning trajectories.

Footnote 6: [https://gutenberg.org](https://gutenberg.org)

Signature Estimation.To estimate the signatures for each word, we sample 100 positive and 100 negative contexts from the BabyLM test set. To ensure fair cross-model comparisons, we use the same test contexts for all models, regardless of training data.

Models.We train GPT-2 from scratch following the training procedure described by radford2019learning. To reduce variance in performance due to random initialization, we train three variations of each model using different random seeds. To compute the reference signatures (\(\widehat{\sigma}_{\mathrm{R+}}\), \(\widehat{\sigma}_{\mathrm{R-}}\), \(\widehat{\sigma}_{\mathrm{R\pm}}\)) we use Llama-3.1-8B7 as the reference distribution \(r\). Full details regarding the hyperparameters, training duration, and loss curves are provided in App. C. As we are interested in analyzing the learning trajectories for models, it is important that they are trained for a reasonable duration. For models trained on BabyLM and CHILDES we apply early stopping, i.e., we choose the best model on a held-out development set, as we found that models eventually overfit. For models trained on Unified we train for \(30{,}000\) steps, or 12 epochs, following [13]. We estimate that Chang and Bergen2022 

their models on about \(1.6\times 10^{9}\) input tokens (counting repetitions).8

Footnote 7: This estimate is based on the reported 100k steps with a batch size of 128 and a context window of 128.

### The Wordbank Corpus

Child \(AoA\) data comes from the North American English portion of the Wordbank database Frank et al. (2017). For each word and month, Wordbank provides the proportion of children in the study that have produced the word by that point. The \(AoA\) is defined as the first month by which at least 50% of children have produced that word Goodman et al. (2008); Braginsky et al. (2016). We exclude words for which we were not able to sample 100 positive context types from the BabyLM dataset, leaving us with 305 words. The words in Wordbank are divided into 4 different **lexical categories**: Nouns (101), Predicates (124), Function Words (45) and Other (49)9. The Predicates category is further divided into Adjectives and Verbs.10

Footnote 8: The total word count across all categories should be 319, but it is actually 305 due to words appearing in multiple categories.

Footnote 10: Some words, e.g., _babysitter_, _doctor_, and _brother_, are categorized as other even though they belong to the category nouns in the category Other. We exclude such misannotated words from our analyses, yielding 262 words in total.

## 6 Examining LM Learning Trajectories

Before quantitatively comparing LM and child word learning trajectories in SS7, we conduct several analyses focusing solely on LM trajectories.

### Case studies

We perform several case studies by inspecting the learning trajectories and \(AoA\) scores for humans and each distributional signature from SS3. We analyze the trajectories and \(AoA\) scores for LMs trained on the Unified dataset for a sample of 8 words: two function words, two nouns, two adjectives, and two verbs. For each category, one word is chosen from the 10 most and 10 least frequent (for the Unified dataset).

Fig. 1 shows the trajectories for these words, and Table A5 gives the \(AoA\) scores. For most signatures, we observe that the higher-frequency word from a category has an earlier \(AoA\) than the corresponding lower-frequency word. We also observe that most signatures yield a wide range of \(AoA\) scores, but others--particularly \(\widehat{\sigma}_{-}\)--show very similar (and late) \(AoA\)s for all words we inspect. Table A6 shows the first and last learned words for each signature. Generally, we find that high-frequency words and function words are learned first.

### Convergence behavior

As we rely on the Cauchy criterion to extract \(AoA\) scores, we now examine how different signatures converge. Fig. 1 shows that the shape of the learning trajectories varies between signatures. Within a given signature, trajectory shapes are internally consistent to varying degrees. As expected, the reference signatures are mostly monotonically

Figure 1: Trajectories for a sample of 8 words for LMs trained on the Unified dataset. We sample one high-frequency (solid line) and one low-frequency (dashed) word from each of the categories: function words, nouns, adjectives, verbs. The \(y\)-axis represents the value of the estimator in all \(\widehat{\sigma}\) plots. For the children, it represents the proportion of them that produced the word.

 

decreasing, indicating that the probability of the word of interest under the LM becomes closer to the ground truth after more iterations. Furthermore, for the corpus-based signatures, \(\widehat{\sigma}_{+}\) trajectories are decreasing, whereas \(\widehat{\sigma}_{-}\) are increasing.11 On the other hand, the intrinsic signatures and \(\widehat{\sigma}_{\pm}\) are not consistently increasing or decreasing.

Footnote 11: One caveat: for the first few training steps, the trajectory sometimes moves in the other direction. Chang and Bergen (2022) observed this phenomenon, showing that the learned distribution initially approximates a uniform distribution followed by the unigram distribution. After this point, the trajectories are largely monotonic.

We compute the \(AoA\) scores for a given signature using a range of values for \(\epsilon\). Fig. A4 shows how many words failed to converge under different thresholds \(\epsilon\). We find that the vast majority of trajectories converge with \(\epsilon=0.15\). For lower values of \(\epsilon\), we see as many as half of all word trajectories failing to converge on the CHILDES dataset. However, on the larger datasets BabyLM and Unified, we observe high rates of convergence across the board. Finally, it is the intrinsic signatures and \(\widehat{\sigma}_{\pm}\) that show the lowest rates of convergence. As discussed above, these are precisely the same signatures that do not have an internally consistent shape. Furthermore, the figures in App. I show correlations for each pair of thresholds. With a few exceptions for extreme values, different thresholds still yield \(AoA\) scores that are highly correlated. Therefore, in all our results (including those discussed above), we apply an intermediate value of \(\epsilon=0.07\).

### Comparing Signatures

Another important question is whether different signatures give similar \(AoA\) scores to each other. App. J shows the correlation matrix of \(AoA\) scores for each signature. First, the correlations are all notably higher for LMs trained on the Unified dataset. Together with the finding that convergence rates are higher for this dataset, this supports the conclusion that \(AoA\) scores become more consistent as training time increases. We find that most pairs of signatures are weakly or negatively correlated with few exceptions. In general, the various positive signatures (\(\widehat{\sigma}_{+}\), \(\widehat{\sigma}_{1+}\), \(\widehat{\sigma}_{\mathrm{R}+}\)) have relatively strong correlations. Across all datasets, the most strongly correlated pair is \(\widehat{\sigma}_{+}\) and \(\widehat{\sigma}_{\mathrm{R}+}\). The negative signatures, but \(\widehat{\sigma}_{1-}\), have weak correlations with other signatures, except for the pair \(\widehat{\sigma}_{-}\), \(\widehat{\sigma}_{\pm}\) which have very similar estimators.

## 7 Human vs. LM Learning Trajectories

We now examine the similarities and differences between word learning in LMs and humans.

### Comparing Human and LM \(AoAs\)

We begin simply by measuring the Pearson correlation between human \(AoA\)s and the LM \(AoA\)s from each signature. These values are plotted in Fig. 2. Overall, we observe very weak or negative correlations. We find that the signature that correlates most changes depending on the datasets, but no correlation exceeds 0.31 (either positive or negative). The strongest positive correlations occur in BabyLM for \(\widehat{\sigma}_{1\pm}\) and \(\widehat{\sigma}_{\mathrm{R}-}\), while the strongest negative correlations are from Unified and BabyLM for the \(\widehat{\sigma}_{1-}\) signature.

### Predicting \(AoAs\) from Features

We now investigate which factors predict human and LM \(AoA\)s, and compare whether these factors have similar effects. Braginsky et al. (2016) identified several interpretable features that predict human \(AoA\)s. Chang and Bergen (2022) previously fit linear models to predict LM \(AoA\)s using these features. We extend this analysis to our set of signatures. Specifically, we take the \(AoA\) scores of children and of each of our signatures as our dependent variables, and, additionally, consider the following predictors12 that Braginsky et al. (2016) studied: (i) **log frequency** with respect to each LM's training dataset for LMs and with respect to CHILDES for children, (ii) **number of characters**, i.e., the number of symbols from \(\Sigma\) in the

Figure 2: Pearson’s correlation coefficients between different signatures and children’s \(AoA\) (C) across three datasets: CHILDES, BabyLM, and Unified.

 

word, (iii) **concreteness** judgments, collected from human subjects by Brysbaert et al. (2014), which indicates the extent to which a word is concrete, measured on a scale from 1 (very abstract) to 5 (very concrete), (iv) **mean length of the utterances (MLU)** with respect to each LM's training dataset for LMs and with respect to CHILDES for children, and (v) **lexical category** noun, predicate, function word, and other, annotated by Frank et al. (2017, 2021).

Do similar factors influence LMs and Children_AoAs?_Regression analyses for LMs trained on Unified and for children are presented in Table 2. For children, the adjusted \(R^{2}\) with all predictors is \(0.417\). The strongest individual predictors of children's \(AoA\) are _concreteness_ and _lexical category_. _Log frequency_ is a notably weak predictor on its own, though it does still contribute significant predictive power when added to a model including all other predictors. These results largely reproduce those of (Braginsky et al., 2016) and Chang and Bergen (2022), the latter of whom reported an adjusted \(R^{2}\) of \(0.43\) for predicting child \(AoA\) from all features using a larger vocabulary of 571 words. In predicting LMs' \(AoA\), we identify two main patterns. First, the signatures \(\widehat{\sigma}_{l\pm}\) and \(\widehat{\sigma}_{-}\) exhibit negligible relationships with any predictor. Second, among the other signatures, _log frequency_ is consistently the most predictive factor, mirroring the findings of Chang and Bergen (2022). The next most predictive factors are _concreteness_ and _lexical category_. See the figures in App. G or visualizations of \(AoA\)s against each predictor. For brevity, we focus on results from the Unified dataset. Across all signatures and datasets, language models (LMs) exhibit the opposite pattern from children when it comes to _log frequency_ and _concreteness_, with more frequent words having a lower \(AoA\). While children tend to acquire concrete words earlier, language models appear to struggle more with processing concrete words and perform better with abstract ones. Furthermore, although children's \(AoA\) does not display a significant correlation with _number of characters_, most LM signatures reveal positive correlations. The exceptions are \(\widehat{\sigma}_{\mathrm{R}-}\) and \(\widehat{\sigma}_{l\pm}\) (for BabyLM and CHILDES), which show slightly negative correlations. Lastly, _MLU_ follows a similar pattern in both children and language models.

Does more developmentally plausible training data result in more human-like learning patterns?From Fig. 2, we observe that the models trained on BabyLM tend to exhibit the most human-like learning trajectories according to some signatures; however, as stated above, LM trajectories are far from human-like across the board. This finding is surprising given that CHILDES, which comes from discourses between caregivers and young children, most closely resembles the input received by the young children studied in Wordbank. However, since all three datasets differ greatly in size, we cannot determine whether this result is due to data domain or dataset size. By analyzing Table A8, we find that there is a positive effect of the training set size on the predictability of \(AoA\). We speculate that this may explain why Chang and Bergen

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multicolumn{2}{c}{**Metadata**} & \multicolumn{4}{c}{**Single-Predictor**} & \multicolumn{2}{c}{**Multi-Predictor**} \\ \(AoA\) **type** & **\#words** & **Log freq.** & **Conc.** & **\#chars** & **MLU** & **Lex. cat.** & **Full** & **Full** \textbackslash Log freq.** \\ \hline _Children_ & _262_ & _0.004_ & _0.26_ & _-0.003_ & _0.032_ & _0.20_ & _0.417_ & _0.363_ \\ \(\widehat{\sigma}_{+}\) & 262 & **0.614** & **0.298** & 0.135 & 0.072 & **0.304** & **0.616** & **0.392** \\ \(\widehat{\sigma}_{-}\) & 251 & 0.063 & 0.012 & 0.023 & 0.0 & 0.047 & 0.083 & 0.065 \\ \(\widehat{\sigma}_{\pm}\) & 245 & 0.542 & 0.265 & **0.142** & 0.04 & 0.294 & 0.546 & 0.379 \\ \(\widehat{\sigma}_{1+}\) & 215 & 0.4 & 0.274 & 0.107 & **0.162** & 0.201 & 0.463 & 0.382 \\ \(\widehat{\sigma}_{1-}\) & 197 & 0.234 & 0.168 & 0.028 & 0.035 & 0.126 & 0.256 & 0.179 \\ \(\widehat{\sigma}_{1\pm}\) & 201 & 0.052 & 0.012 & 0.005 & 0.006 & 0.05 & 0.063 & 0.05 \\ \(\widehat{\sigma}_{\mathrm{R}+}\) & 262 & 0.572 & 0.295 & 0.118 & 0.088 & 0.296 & 0.582 & 0.377 \\ \(\widehat{\sigma}_{\mathrm{R}-}\) & 262 & 0.013 & 0.0 & 0.001 & 0.007 & 0.003 & 0.033 & 0.013 \\ \(\widehat{\sigma}_{\mathrm{R}\pm}\) & 256 & 0.292 & 0.159 & 0.083 & 0.04 & 0.122 & 0.3 & 0.2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of model results for Child \(AoA\) and LMs trained on the Unified dataset. Note: _lexical category_ does not contain the category other which includes words that could be assigned to nouns, predicates, or function words.

 

(2022), who trained models on much larger datasets than ours, reported higher predictability for model \(AoA\) scores (for \(\widehat{\sigma}_{+}\)). Additionally, we note that in the CHILDES dataset, _log frequency_ is not significantly more predictive than other factors, in contrast to other datasets. Overall, the results do not exhibit any notable human-like patterns.

## 8 Discussion and Conclusion

Our main objective was to explore the space of distributional signatures of words more comprehensively, with an application to understanding word learning in LMs. We showed that the distributional test adopted by Chang and Bergen (2022) and Portellance et al. (2023) can be viewed as an estimator of a more general distributional signature. This insight also enabled us to define a broader family of signatures that follow a clear typology. However, the question remains: which of these evaluations should be the focus of researchers interested in studying word learning in LMs? One of our key findings in SS6.3 is that many of these signatures are complementary. This is true with respect to children's \(AoAs\) as well as in comparison to each other. Arguably, considering both positive and negative contexts provides a more complete picture of the LM's distributional knowledge, and comparing the LM's distribution against an LLM allows the signature to better reflect the gradient of the ground truth distribution, which is not directly observable. Nonetheless, each signature we propose has a clear interpretation and may be useful for specific applications, though extracting usable \(AoA\) scores is not always feasible.

In SS7 we found that we could not predict children's \(AoA\)s well from any of our proposed distributional signatures. This result might seem somewhat surprising in light of Portelance et al.'s (2023) finding that LM surprisal improves predictions of children's \(AoA\)s. However, we note that that work uses \(\widehat{\sigma}_{+}\) at the end of training as a predictor of \(AoA\), rather than the \(AoA\) of the model as determined by a specific distributional signature. Our results do further corroborate Chang and Bergen's (2022) conclusions on this question, and significantly expand them to a wider variety of signatures. Additionally, our findings contribute to a growing body of work finding specific differences between the language learning patterns of humans and LMs in other domains (e.g., Evanson et al., 2023; Constantinescu et al., 2024). On the other hand, Zhuang et al. (2024b, a) show that multimodal LMs can exhibit more human-like learning trajectories and also introduce a novel training objective that further improves human-likeness.

Future work should apply our distributional tests to these and other potentially more human-like training procedures. Besides learning in a world grounded in sensory experience, children also learn through interaction both with the physical world and with other agents Clark (2018); Nikolaus and Fourtassi (2023). Moreover, unlike LMs, children have constraints on production, going through one-word and two-word utterance phases Bloom (1970). These factors no doubt influence the kinds of words children use early in development and may account for the precedence of concrete words. There are only a few examples of training regimes for LMs inspired by interaction Lazaridou et al. (2020); Nikolaus and Fourtassi (2021); Ma et al. (2024). Furthermore, the reliance on stochastic gradient descent and cross-entropy loss likely skew learning trajectories in LMs in ways that are not entirely human-like. There are many opportunities to explore more human-like LM training, and we expect that word learning will be an important evaluation of human likeness as these are explored.

Having better mapped out the space of distributional signatures of lexical knowledge, our work paves the way for comparing trajectories of language models and humans. Our findings provide strong empirical support that there are large differences between how human and artificial language learners develop throughout learning and draw attention to the fact that there is significant work to be done to explore pre-training methods and datasets that result in more developmentally plausible language models.

## 9 Limitations

Our study has several limitations. First, while we are interested in the possibility that LMs can be used as cognitive models and we attempt to use developmentsl plausible data, our LMs are not trained in a way that is maximally similar to how humans learn. They lack exposure to speech, grounding, and interaction with other agents, all of which may significantly influence word learning. Second, while our proposed true and reference signatures are weighted by a distribution \(\overrightarrow{p_{\kappa}}\), we only estimate this distribution using Monte Carlo estimation. Future work should explore whether alterna 

tive estimation techniques, such as those based on LLMs, yield qualitatively different results. Third, the specifics of our findings may be sensitive to our training setup. Future work should examine whether different pre-training settings yield qualitatively different results, i.e., whether our findings are robust across various setups. Finally, our study focuses on extracting \(AoA\)s from learning trajectories, but \(AoA\) is just one statistic that can be extracted from the learning trajectory.

## Acknowledgments

We thank the reviewers and ACs for ARR whose detailed feedback led to many improvements in our work. AW was supported by an ETH Postdoctoral Fellowship.

## References

* Berger et al. (2022) Uri Berger, Gabriel Stanovsky, Omri Abend, and Lea Frermann. 2022. A computational acquisition model for multimodal word categorization. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3819-3835.
* Bloom (1970) Lois Bloom. 1970. _Language Development: Form and Function in Emerging Grammars_. MIT Press.
* Braginsky et al. (2019) Mika Braginsky, Daniel Yurovsky, Virginia A. Marchman, and Michael C. Frank. 2019. Consistency and variability in children's word learning across languages. _Open Mind: Discoveries in Cognitive Science_, 3:52-67.
* Braginsky et al. (2016) Mika Braginsky, Daniel Yurovsky, Virginia A. Marchman, and Mike C. Frank. 2016. From uh-oh to tomorrow: Predicting age of acquisition for early words across languages. _Proceedings of the Annual Meeting of the Cognitive Science Society_, 38:1691-1696.
* Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 136-145, Jeju Island, Korea. Association for Computational Linguistics.
* Brysbaert et al. (2014) Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Carorectness ratings for 40 thousand generally known English word lemmas. _Behavior Research Methods_, 46(3):904-911.
* Buchanan et al. (2019) Erin M. Buchanan, K. D. Valentine, and Nicholas P. Maxwell. 2019. English semantic feature production norms: An extended database of 4436 concepts. _Behavior Research Methods_, 51(4):1849-1863.
* Chang and Bergen (2022) Tyler A. Chang and Benjamin K. Bergen. 2022. Word acquisition in neural language models. _Transactions of the Association for Computational Linguistics_, 10:1-16.
* Chang et al. (2024) Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. 2024. Characterizing learning curves during language model pre-training: Learning, forgetting, and stability. _Transactions of the Association for Computational Linguistics_, 12:1346-1362.
* Choshen et al. (2022) Leshem Choshen, Ryan Cotterll, Michael Y. Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. [Call for papers] the 2nd babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. _Preprint_, arXiv:2404.06214.
* Choshen et al. (2022) Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. 2022. The grammar-learning trajectories of neural language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, volume 1, pages 8281-8297.
* Clark (2018) Eve V. Clark. 2018. Conversation and Language Acquisition: A Pragmatic Approach. _Language Learning and Development_, 14(3):170-185.
* Constantinescu et al. (2024) Ionut Constantinescu, Tiago Pimentel, Ryan Colterell, and Alex Warstadt. 2024. Investigating critical period effects in language acquisition through neural language models. _Transactions of the Association for Computational Linguistics_.
* Datta et al. (2023) Arghya Datta, Subhrangshu Nandi, Jingcheng Xu, Greg Ver Steeg, He Xie, Anoop Kumar, and Aram Galstyan. 2023. Measuring and mitigating local instability in deep neural networks. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2810-2823, Toronto, Canada. Association for Computational Linguistics.
* Deerwester et al. (1990) Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. _Journal of the American Society for Information Science_, 41(6):391-407.
* Dupoux (2018) Emmanuel Dupoux. 2018. Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner. _Cognition_, 173:43-59.
* Evanson et al. (2023) Linnea Evanson, Yair Lakretz, and Jean Remi King. 2023. Language acquisition: do children and language models follow similar learning stages? In _Findings of the Association for Computational Linguistics_, pages 12205-12218.
* Fenson et al. (2013) Larry Fenson, Virginia Marchman, Donna Thal, Phillip Dale, Steven Reznick, and Elizabeth Bates. 2013. _MacArthur-Bates Communicative Development Inventories_. Springer 

* Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytant Ruppin. 2001. Placing Search in Context: The Concept Revisited. In _Proceedings of the 10th international conference on World Wide Web_, pages 406-414.
* Firth (1957) J. R. Firth. 1957. A synopsis of linguistic theory, 1930-1955. _Studies in Linguistic Analysis_.
* Frank et al. (2021) Michael C. Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A. Marchman. 2021. _Variability and consistency in early language learning: The Wordbank project_. Variability and consistency in early language learning: The Wordbank project. The MIT Press.
* Frank et al. (2017) Micheal C. Frank, Mike Braginsky, Daniel Yurovsky, and Virginia A. Marchman. 2017. Wordbank: an open repository for developmental vocabulary data. _Journal of Child Language_, 44(3):677-694.
* Gerlach and Font-Clos (2020) Martin Gerlach and Francesce Font-Clos. 2020. A standardized project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics. _Entropy_, 22(1).
* Gerz et al. (2016) Daniela Gerz, Ivan Vulic, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2173-2182, Austin, Texas. Association for Computational Linguistics.
* Gilkerson et al. (2017) Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kimbrough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. _American Journal of Speech-Language Pathology_, 26(2):248-265.
* Goodman et al. (2008) Judith C. Goodman, Philip S. Dale, and Ping Li. 2008. Does frequency count? parental input and the acquisition of vocabulary. _Journal of Child Language_, 35(3):515-531.
* Hale (2001) John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In _Second Meeting of the North American Chapter of the Association for Computational Linguistics_.
* Harris (1954) Zellig Harris. 1954. Distributional structure. _Word_, 10(2-3):146-162.

* Hermans et al. (2017) Joeri R. Hermans, Gerasimos Spanakis, and Rico Mockel. 2017. Accumulated gradient normalization. In _Proceedings of the Ninth Asian Conference on Machine Learning_, volume 77 of _Proceedings of Machine Learning Research_, pages 439-454, Yonsei University, Seoul, Republic of Korea. PMLR.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Hofmann (1999) Thomas Hofmann. 1999. Probabilistic latent semantic analysis. In _UAI_, volume 99, pages 289-296.
* Landauer and Dumais (1997) Thomas K. Landauer and Susan T. Dumais. 1997. A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. _Psychological Review_, 104(2):211.
* Lavechin et al. (2022) Marvin Lavechin, Maureen De Seyssel, Lucas Gautheron, Emmanuel Dupoux, and Alejandrina Cristia. 2022. Reverse engineering language acquisition with child-centered long-form recordings. _Annual Review of Linguistics_, 8(1):389-407.
* Lazaridou et al. (2020) Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7663-7674, Online. Association for Computational Linguistics.
* Lencic (2008) Alessandro Lencic. 2008. Distributional semantics in linguistic and cognitive research. _Italian Journal of Linguistics_, 20(1):1-31.
* Levy (2008) Roger Levy. 2008. Expectation-based syntactic comprehension. _Cognition_, 106(3):1126-1177.
* Linzen (2019) Tal Linzen. 2019. What can linguistics and deep learning contribute to each other? _Language_, 95(1):e99-e108.
* Lison and Tiedemann (2016) Pierre Lison and Jorg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation_, pages 923-929.
* Ma et al. (2024) Ziqiao Ma, Zeku Wang, and Joyce Chai. 2024. Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations. _arXiv preprint_. ArXiv:2405.13828 [cs].
* MacWhinney (2000) Brian MacWhinney. 2000. _The CHILDES Project: Tools for analyzing talk: The database, Vol. 2, 3rd ed._ Lawrence Erlbaum Associates Publishers.
* Marantz (2001) Alec Marantz. 2001. Words. _WCCFL XX Handout, USC_.
* Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted Syntactic Evaluation of Language Models. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202.
* Meta (2024) AI @ Meta. 2024. The Llama 3 herd of models. _Preprint_, arXiv:2407.21783 

* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems_, volume 26.
* Nikolaus and Fourtassi (2021a) Mitja Nikolaus and Abdellah Fourtassi. 2021a. Evaluating the acquisition of semantic knowledge from cross-situational learning in artificial neural networks. In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_, pages 200-210. Association for Computational Linguistics.
* Nikolaus and Fourtassi (2021b) Mitja Nikolaus and Abdellah Fourtassi. 2021b. Modeling the interaction between perception-based and production-based learning in children's early acquisition of semantic knowledge. In _Proceedings of the 25th conference on computational natural language learning_, pages 391-407, Online. Association for Computational Linguistics.
* Nikolaus and Fourtassi (2023) Mitja Nikolaus and Abdellah Fourtassi. 2023. Communicative Feedback in language acquisition. _New Ideas in Psychology_, 68:100985.
* Opedal et al. (2024) Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, and Ethan Wilcox. 2024. On the role of context in reading time prediction. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 3042-3058.
* OpenAI (2022) OpenAI. 2022. ChatGPT: Optimizing language models for dialogue.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543.
* Portelance et al. (2023) Eva Portelance, Yuguang Duan, Michael C. Frank, and Gary Lupyan. 2023. Predicting age of acquisition for children's Early vocabulary in five languages using language model surprisal. _Cognitive Science_, 47(9).
* Portelance et al. (2024) Eva Portelance, Michael C. Frank, and Dan Jurafsky. 2024. Learning the meanings of function words from grounded language using a visual question answering model. _Cognitive Science_, 48(5).
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* V)_, pages 69-79. The COLING 2016 Organizing Committee.
* Stolcke et al. (2000) Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. _Computational Linguistics_, 26(3):339-373.
* Wong et al. (2024) Wai Keen Wong, Wentao Wang, A. Emin Orhan, and Brenden M. Lake. 2024. Grounded language acquisition through the eyes and ears of a single child. _Science_, 383(6682):504-511.
* Warstadt and Bowman (2022) Alex Warstadt and Samuel R Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In _Algebraic Structures in Natural Language_, pages 17-60. CRC Press. Publisher: CRC Press.
* Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohanney, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. _Transactions of the Association for Computational Linguistics_, 8:377-392.
* Wittgenstein (1953) Ludwig Wittgenstein. 1953. _Philosophical investigations_. Wiley-Blackwell.
* Zhuang et al. (2024a) Chengxu Zhuang, Evelina Fedorenko, and Jacob Andreras. 2024a. Lexicon-level contrastive visual-grounding improves language modeling. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 231-247.
* Zhuang et al. (2024b) Chengxu Zhuang, Evelina Fedorenko, and Jacob Andreras. 2024b. Visual grounding helps learn word meanings in low-data regimes. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1311-1329.

 

The Intrinsic Signature

We now overview the intrinsic signature in more detail, which was omitted from SS3.

### An Intrinsic Metric

We develop an intrinsic metric, i.e., a metric that does not relay, in expectation, on the true language model \(p\). Thus, we consider the following information-theoretic quantity that resembles Eq.6, but where the expectation is taken with respect to the model itself:

\[\sigma_{1+}\stackrel{{\mbox{\tiny def}}}{{=}}-\sum_{\mathbf{c}\in \Sigma^{*}}\overrightarrow{q_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w})\log\overrightarrow{ q}(\mathbf{w}~{}|~{}\mathbf{c}). \tag{1}\]

In contrast to Chang and Bergen's (2022) distribution signature, Eq.1 is not grounded in an external language model. Thus, it measures a notion of knowledge internal to the language model itself. We can also, by analogy to Eq.9, define an intrinsic metric that considers just negative contexts

\[\sigma_{1-}\stackrel{{\mbox{\tiny def}}}{{=}}-\sum_{\mathbf{c}\in \Sigma^{*}}\overrightarrow{q_{\kappa}}(\mathbf{c}~{}|~{}\neg\mathbf{w})\log \overrightarrow{q}(\mathbf{w}~{}|~{}\mathbf{c}), \tag{2}\]

and one that considers all contexts

\[\sigma_{1\pm}\stackrel{{\mbox{\tiny def}}}{{=}}-\sum_{\mathbf{c}\in \Sigma^{*}}\overrightarrow{q_{\kappa}}(\mathbf{c})\log\overrightarrow{q}(\mathbf{ w}~{}|~{}\mathbf{c}). \tag{3}\]

### A Practical Estimator

We now discuss a scheme to estimate Eq.1. First, we note that, by Bayes' rule, we have

\[\overrightarrow{q_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w})=\frac{\overrightarrow{q}( \mathbf{w}~{}|~{}\mathbf{c})\overrightarrow{q}(\mathbf{c})}{\sum_{\mathbf{c}\in\Sigma^{*}} \overrightarrow{q}(\mathbf{w}~{}|~{}\mathbf{c})\overrightarrow{q}(\mathbf{c})}. \tag{4}\]

Instead, we consider the following approximation. Given a bag of contexts \(\mathcal{C}=\nmid c^{(m)}\rangle_{m=1}^{M}\) that proceed a word \(\mathbf{w}\), we construct the following empirical approximation

\[\widetilde{q}_{\kappa}(\mathbf{c}~{}|~{}\mathbf{w})=\frac{\mathbbm{1}\{\mathbf{c}\in \mathcal{C}\}\overrightarrow{q}(\mathbf{w}~{}|~{}\mathbf{c})\overrightarrow{q_{\kappa }}(\mathbf{c})}{\sum_{m=1}^{M}\overrightarrow{q}(\mathbf{w}~{}|~{}\mathbf{c}^{(m)}) \overrightarrow{q_{\kappa}}(\mathbf{c}^{(m)})}. \tag{5}\]

Plugging Eq.5 into Eq.1, we arrive at

\[\widehat{\sigma}_{1+}\stackrel{{\mbox{\tiny def}}}{{=}}-\sum_{ m=1}^{M}\widetilde{q}_{\kappa}(\mathbf{c}^{(m)}~{}|~{}\mathbf{w})\log\overrightarrow{q}( \mathbf{w}~{}|~{}\mathbf{c}^{(m)}), \tag{6}\]

In the limiting case, i.e., when \(\mathcal{C}\) includes all of \(\Sigma^{*}\), we have \(\widehat{\sigma}_{1+}\rightarrow\sigma_{1+}\). Note that Eq.6 is not a standard Monte Carlo estimator as the contexts \(\mathbf{c}^{(m)}\) may not have been drawn from \(\overrightarrow{q_{\kappa}}(\cdot~{}|~{}\mathbf{w})\), but it is still consistent. An analogous estimator can be derived for Eq.2 and Eq.3.

Appendix B The Signature \(\boldsymbol{\sigma_{\mathrm{R}+}},\boldsymbol{\sigma_{\mathrm{R}-}}\) and \(\boldsymbol{\sigma_{\mathrm{R}\pm}}\) are Distance Metrics

The **reference** signatures as introduced in SS3 can be easily shown to be distance metrics. Let \(x_{c}\stackrel{{\mbox{\tiny def}}}{{=}}\log\overrightarrow{p_{ \kappa}}(\mathbf{c}~{}|~{}\mathbf{w})\) and \(y_{c}\stackrel{{\mbox{\tiny def}}}{{=}}\log\overrightarrow{r}(\mathbf{ w}~{}|~{}\mathbf{c})\), we can rewrite the signatures as follows:

\[\sigma_{\mathrm{R}+} =\sum_{\mathbf{c}\in\Sigma^{*}}\overrightarrow{p_{\kappa}}(\mathbf{c}~{}| ~{}\mathbf{w})\left|x_{c}-y_{c}\right| \tag{7a}\] \[\sigma_{\mathrm{R}-} =\sum_{\mathbf{c}\in\Sigma^{*}}\overrightarrow{p_{\kappa}}(\mathbf{c}~{}| ~{}\neg\mathbf{w})\left|x_{c}-y_{c}\right|\] (7b) \[\sigma_{\mathrm{R}\pm} =\sum_{\mathbf{c}\in\Sigma^{*}}\overrightarrow{p_{\kappa}}(\mathbf{c}) \left|x_{c}-y_{c}\right|. \tag{7c}\]

Because \(\overrightarrow{p_{\kappa}}(\mathbf{c}~{}|~{}\mathbf{w}),\overrightarrow{p_{\kappa}}( \mathbf{c}~{}|~{}\neg\mathbf{w})\) and \(\overrightarrow{p_{\kappa}}(\mathbf{c})\) are all greater than zero, the expressions above represent weighted Manhattan distances, which is a known distance metric 

Training Details

The training was conducted in parallel across 8 GPUs, with gradient accumulation (Hermans et al., 2017) set to 16 and a batch size per device of 4. As a result, our model was trained with an effective batch size of 512.

We saved the checkpoints used for our analysis at increasing intervals throughout the training. Specifically, we saved checkpoints

* every 50 steps for integers in \((0,1000]\);
* every 200 steps for integers in \((1000,10000]\);
* every 500 steps for integers in \((10000,30000]\).

## Appendix D Convergence with the Cauchy Criterion

To judge the convergence of learning trajectories that are non-monotonic and may not have a well-formed shape, we develop a novel technique based on the idea of a Cauchy sequence. Let \(\sigma({\mathbf{w}},t)\) the value that the signature of the word \({\mathbf{w}}\) assumes at time-step \(t\). For a fixed tolerance parameter \(\epsilon>0\), the age of acquisition \(AoA\) is defined as

\[AoA=\alpha\left(\sigma,{\mathbf{w}}\right)=\operatornamewithlimits{argmin}_{t\in \{1,...,T\}}\left(\max_{s,s^{\prime}\in\{t,...,T\}}|\sigma({\mathbf{w}},s)-\sigma ({\mathbf{w}},s^{\prime})|<\epsilon\right). \tag{8}\]

This definition mirrors the definition of the convergence of a Cauchy sequence, albeit for a finite sequence. However, because \(T\) is finite, for small enough \(\epsilon\), we do not, in general, observe true convergence in the analytic sense. Thus, the tolerance parameter \(\epsilon\) is best viewed as a hyperparameter, and our findings are dependent on the choice of \(\epsilon\). However, given that nearly all learning algorithms are analyzed by letting 

\(T\rightarrow\infty\), there is a sense in which our definition of AoA is well-founded. Specifically, if we assume the convergence of the learning algorithm as \(T\rightarrow\infty\) implies the convergence of \(\sigma\), then, for every \(\epsilon>0\), there exists a number of epochs such that we will achieve \(\alpha\left(\sigma,\mathbf{w}\right)\).

Fig. A4 shows the percentage of words that did not reach convergence for various \(\epsilon\) values. A word is marked as non-converged if \(\sigma\) failed to converge for even a single seed.

## Appendix E Case Studies

Table A5 presents the \(AoA\) for each word illustrated in Fig. 1. The \(AoA\) values for the signatures were obtained from the model trained on the Unified dataset, using seed 42, and extracted using \(\epsilon=0.07\).

 

First and Last Acquired Words

Table A6 reports the first 10 and the last 10 words that were acquired according to each signature. The words refer to the model trained on the Unified dataset, using seed 42 and extracted using \(\epsilon\) = 0.07.

## Appendix G _AoA_ vs Predictors

In the following subsections, we show how log frequency, MLU, number of characters, and concreteness each influence \(\sigma\)'s \(AoA\) across different datasets. Each \(AoA\) value presented in the plots represents the average \(AoA\) across multiple seeds, extracted with \(\epsilon\) = 0.07. Only words that achieved convergence across all seeds were included in the analysis.
 

* [10] M. C. Dickson, J. R. Kneib, and A. S. Thorpe, "Theoretical Population of the Universe: The Role of Life in the Early Universe," _Astrophys. J. Suppl._**453**(2), 79-86, 2000.
* [11] G. P. Salamon, E. L. Smith, and T. W. Taylor, "A New Approach to the Evolution of the Universe," _Monthly Notices of the Royal Astronomical Society_**46**(1), 1-10, 1995.
* [12] H. B. Nielsen, "The Evolution of the Universe," _Annals Phys._**108**(1), 1-11, 1975.
* [13] H. B. Nielsen, "The Evolution of the Universe," _Phys. Rev. Lett._**100**(1), 012001.

[MISSING_PAGE_POST]

 



### BabyLM

[FIGURE:A6.F1][ENDFIGU



### Unified


* [10] M. C. Dickson, J. R. Friedman, and A. S. G. Wilson, _The New Journal of Physics: Theory and Experiment_, vol. 34, no. 2, pp. 57-68, 1970.
* [11] T. W. Fulton, _The Annals of Mathematical Statistics_, vol. 1, p. 103, 1970.
* [12] H. B. Kramers, _Statistical Mechanics_, vol. 1, p. 103, 1970.
* [13] L. P. Landau, _Methods in modern physics_, vol. 1, p. 103, 1970.
* [14] V. I. Arnold, _A Course on Statistical Mechanics_, vol. 1, p. 103, 1970.

[MISSING_PAGE_POST]

 



## Appendix H Regression analysis

```
full<-paste(predictors,collapse="+") reduced<-paste(original_predictors[-1],collapse="+") vif_values<-vif(lm(AoA-full,data=data)) if(max(virf_values)>5){ print("Multicollinearity_detected\n") } predictors<-c("log_frequency","concreteness","n_chars","mlu","lexical_class") for(iin1:length(predictors)){ formula<-paste("AoA_~",predictors[[i]]) model<-lm(formula,data=data) cat(paste(predictors[[ii]],"Adjust_Rsquared:",summary(model)$adj.r.squared)) } m_full<-lm(AoA~full,data=data) m_reduced<-ln(AoA~reduced,data=data) cat("Full_model_Adjust_Rsquared:",summary(m_full)$adj.r.squared cat("Reduced_model_Adjust_Rsquared:",summary(m_reduced)$adj.r.squared
```

Listing 1: simplified R code for the regression analysis.

In the following regressions, we denote _log frequency_ as **LF**, _concreteness_ as **Co**, _number of characters_ as **NC**, _mean length of utterances_ as **MLU**, and _lexical category_ as **LC**. No VIF value exceeds 5, indicating that while some multicollinearity may be present, it is not severe.




### Regression on Language Model Learning Data

The table below reports the Adjusted \(R^{2}\) values for each predictor across all datasets and signatures, as introduced in App. G. Consistent with previous analyses, the \(AoA\) values were computed using a \(\epsilon\) = 0.07. The table shows the number of words included in the regression analysis, counting only those that remained after outlier removal and successfully achieving convergence across all three seeds. Among all the signatures, \(\widehat{\sigma}_{+}\) is the one that demonstrates the strongest predictive power across nearly all predictors.



The Impact of Convergence Thresholds on AoA Extraction

The Pearson correlations reported in Fig. 15 illustrate how the \(AoA\) values extracted using varying \(\epsilon\) correlate. This analysis aims to determine whether the choice of \(\epsilon\) significantly impacts the results. As discussed in SS6.2, with few exceptions, the results across different \(\epsilon\) values show high correlations. Therefore, our analysis will remain consistent regardless of the choice of \(\epsilon\).

 

Correlation of Various \(AoA\)

The figures in this section show how each signature's \(AoA\) values correlate with one another across each model trained with different datasets. As a result, the dataset itself influences the correlation patterns among the different signatures. For example, the Unified dataset displays only positive correlations, whereas Childes and BabyLM datasets exhibit negative correlations for \(\widehat{\sigma}_{1\pm}\) and \(\widehat{\sigma}_{\mathrm{R-}}\). The \(AoA\) values were extracted using \(\epsilon\) = 0.07.