Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
Authors: James A. Michaelov
Roger P. Levy
Ben Bergen
Venue: NeurIPS 2025 poster ()
Keywords: language models
training dynamics
n-grams
semantic similarity
heuristics
shortcuts
Primary area: deep_learning
TL;DR: Transformer, Mamba, and RWKV language models show consistent patterns of change in behavior over the course of training

Abstract: We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns of change in their behavior over the course of pretraining. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the $n$-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all language models, with their predicted probabilities for words overfitting to those words' $n$-gram probabilities for increasing $n$ over the course of training. Taken together, these results suggest that learning in neural language models may follow a similar trajectory irrespective of model details.

Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
James A. Michaeloy Roger Levy', Benjamin K. Bergen
Department of Brain and Cognitive Sciences, MIT MIT Libraries CREOS
Deparmtent of Cognitive Science, UCSD
{ a www.rrpeeeitt..ccomo
babergenQucsd.edi
Abstract
We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWeb'Tes vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns change in their behavior over the course of pretraining. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of given word, the n-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases all language models, with their predicted probabilities for words overfitting to those words' n-gram probabilities for increasing n over the course of training. Taken together, these results suggest that learning in neural language models may follow a similar trajectory irrespective of model details.
Introduction
Language models are complex systems that ostensibly exhibit emergent behaviors (Anderson 1972
Nicolis and Prigogine 1977 Laughlin and Pines] 2000 (oConnor 2021 Wei et al., 2022a). Trained only to predict the next word in a sequence given a context, language models learn to generate grammatical sentences, make predictions in line with real-world knowledge, and sodd-give the right prompt hompt--aswer answer questions and exhibit reasoning-like behavior, even without finetuning (Linzen el al.
2016; Wei et al. [a22bba]] Biderman et al., 2023b Hu and Frank 2024 OLMo Team et al. 2025 How do they get there?
While language model learning involves gradual change by some metrics (Schaeffer et al. 2023). researchers have identified sudden shifts model behavior (Du al. 2024). as well as precipitous
changes in model subnetworks that can drastically alter behavior (O (Olsson et al. 2022, Chen et al. 2024a). However, these analyses generally focus on specific, targeted behaviors or sub-networks. We focus more broadly on whether it is possible to characterize the overall behavior of models and how this changes over the course of training. We draw on two main lines of research. The first demonstrates that language model predictions are often sensitive to superficial properties of their input and output, such as training data, frequency of tokens, token co-occurrences, grammatical structures, tasks, and specific facts about the world; the baseline probability of input and output sequences and subsequences; and the degree of semantic similarity between an output word and its context (Forbes et al. 2019: Wei et al. 2021 Michaclov and Bergen| 2022 McCoy et al. 2024 Chang and Bergen 2024). The second Is previous work suggesting that language models overfit n-grams of increasing
the course of training (Karpathy al. 2016 Chang al.) 2024).
These two lines converge to suggest that much of language model behavior can be characterized by relatively simple heuristics.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We focus on three frequency, n n-gram probability, and semantic similarity. We ask to what extent they explain language model behavior at any point in time over the course of training, and how this changes. Our key contributions are the following:
1. We train and release the Parallel architecture (Parc) language models. These are made up of 1,314 language model checkpoints: seeds each of models of the Pythia (Biderman et al.) (0233b), Mamba-1 (Gu and Dao 2024), and RWKV-4 (Peng et al., (023) architectures, with each seed trained on the same Wexxxt (Gokaslan and Cohen| (019) data, and each with 73 checkpoints taken over the course of training. As far as we are aware, these are the first available checkpointed Mamba and RWKV models.
2. We construct and release the Natural Words Context (NaWoCo) dataset, an evaluation set of over 150,000 words in natural sentence contexts extracted from the Wen corpus (Penedo et al., 2024).
3. We show that three simple heuristics heuristiss-frequennyy (i.e., unigram probability). n-gram prob- ability, and semantic similarity to the context-explain up to 98% of the variance in the probabilities assigned to words in this (decontaminated) dataset by our models, as well as the Pythia (Biderman al. 2023b van der Wal et al. 2024 and Open-GPT2 (Karamcheti et al. 2021) language models.
4. We demonstrate that all the models tested (no matter the size, architecture, or training data) overfit to n-grams of increasing over the course of pretraining, and show correlation with semantic similarity after the first 10-100 steps.
2 Related Work
2.1 n-gram-like prediction in language models
Neural language models learn representations that enable them to predict the next word in the sequence based on generalizations rather than simply storing counts of sequences like n-gram models (Markov 1913; [hannon 1948 Jelinek et al., 1975, Baker 1975). However, there is evidence that
both RNNs and transformers do still calculate the n n-gram probabilities of words. Voita et al. (2024). for example, identify neurons in the OPT models (Zhang et al. 2022) that appear to be sensitive to
restricted number of specific n-gram-that is, they appear to be 'dedicated' (Voita et al. 2024) to
these n--ramms- which more numerous in larger models. In addition, in contemporaneous work,
Chang and Bergen (2025) identify bigram circuits in the Pythia models (Biderman et al. 2023b)
Behaviorally, the probabilities calculated by neural language models-bothh recurrent neural networks and transformers-baae been found to be highly correlated with n-gram probabilities, especially early in training (Karpathy et al.) 2016: (un and Lu 2022; Chang and Bergen 2022 Choshen et al. 2022
Bietti et al. 2023] Voita et al. 20044 Akyürek et al. [224|| 2024; Chang et al. 2024: Belrose et al. 2024 Wang et al. (025). A recent study by Chang et al. (2024). 4) for example, found that over the course of pretraining, the probabilities calculated by 5 random seeds of a 11MM-parametr GPT-2 model become differentially correlated with n-gram probabilities (calculated
the same training data) of increasing n over the course of training- that is, they initially become more correlated with unigram probability (i.e. frequency) n-grams of n >> 1, then become bigram probability than other n-grams, and so on until at least n = 5. In a related line of research, Nguyen (2024) found that for given token prediction in test corpus, is possible to construct rule based on a combination of n-grams in the training corpus of a language model that can make the same top prediction as the model with accuracy of up 68-79%, depending on dataset and the procedure used to calculate the optimal rule for each token.
strongly correlated with
2.2 Similarity-based prediction in language models
The extent to which a word is similar to words in its context has also been shown have an impact on the predictions of language models (Kassner and Schütze, 2020 Misra et al. 2020 Michaelov et al. 2021 2024). One example of this is the finding that lexical semantic priming occurs in language models Misra et al. (2020) find that prepending the word 'airplane" to 'I want to become a [MASK]. to create 'airplane. want to become a [MASK]: increases the probability BERT assigns to 'pilot' being the masked token relative to both the original sentence and the sentence with different

---

prepended word such as 'table'. In less targeted form of analysis, Michaelov et al. (022)) find that GPT-2 surprisal (negative probabiiltyy has Pearson correlation similarity between the language model's static embedding of the same word and the mean of the embeddings of the words the context; while Michaelov et al. (022) find that GPT-3 (Brown et al. 2020) surprisal is correlated with the same similarity metric calculated using Glo (Pennington ct al. 2014) (r = -0.46) and fastText (Grave et al., 2018) (r = -0.61) word embeddings. Thus, while there is thus far no direct evidence that contextual semantic similarity plays an explicit causal role in language model predictions, there is at least strong evidence that it correlates with them.
-0.48 with the cosine
3 Experiment 1: Correlations Between Language Model Log-Probabilities and Heuristies
In our first experiment, we investigate the extent to which language model probability correlates with n-gram probability and contextual semantic similarity over the course of pretraining. Going beyond previous work (e.g.. Chang and Bergen| 2022; Choshen al. 2022, Chang and Bergen 2024, Nguyen| 2024 Belrose et al. 2024), we characterize exactly to what extent each n-gram log-probability for n E (1,2,3, 4, 5) is correlated with the log-probability assigned to given word by language models of different architectures over the course of training. We also carry out the same analysis for semantic similarity as calculated using fastText word embeddings (Bojanowski et al. 2017; Grave et al. 2018). We carry out our analyses on a range of language models, which vary in architecture, size, and training dataset. All code, data, analyses, and models are provided in the following repository: https: / /tth//gttt.....// com/ jmi chaelov 11m sattaaviorl--phasse
3.1 Method
3.1.1
Language Models
We carry out our analyses on two sets of pretrained language models with checkpoints taken over the course of their training. The first set were the Pythia models (14M-12B parameters: Biderman itt al. 2023b) including the additional PolyPythia seeds (9 of each of the 14M-410M parameter models; van der Wal et al. (024). all of which were trained on The Pile (Gao et al.) 2020). The second were the GPT-2 models (5 seeds each with 117M and 345M parameters, of which we used 4; Appendix A) trained part of the Mistral project (Karamcheti et al. 2021 on the OpenWebTer. (Gokaslan and Cohen 2019) corpus, henceforth the OPen-GPT2 models.
We also trained an additional 18 language models. Following the approach taken for the Open-GPT2 models (Karamcheti et al. 202T), we trained all models on 1024-token sequences of OpenWehText WebText with batch size of 512; though we train each for only 4,000 steps. In order to expand our analyses beyond transformers, we use the same 6 random seeds to train models with three architectures in parallel (in the sense that they encounter the same sequences at the same steps), using the same tokenizer for all models. The three architectures were the Pythia (Biderman et al. 2023b) transformer architecture, the Mamba-1 state-space model architecture (Gu and Dao (0224) and the RWKV-4 architecture, which is a modern recurrent neural network architecture with parallelizable training (Peng et al. (023). We henceforth refer to these as the Pare-Pythia (160M), Pare-Mamba (130M), and Pa Pare- RWKV (169M) models, respectively.
Considering all models, seeds, and checkpoints, our analysis encompasses a total of 1,418 model instances. We provide the full code for training and running the models, and further details of all models used in Appendix A
3.1.2 n-gram Log-Probability
To investigate the extent to which each language models' predictions match given n-grams, we calculate the n-gram probability of the same words in context for n E {1, 2, 3, 5). We calculate n-grams using the same data as the model itself was trained on-that is, for the Pythia models, we calculate n-grams in The Pile, and for the OpenWebText models (Open-GPT2, Parc-Pythia Pare-RWKV, and Parc-Mamba), we calculate n-grams in OpenWeb Because the pretrained models have different tokenizers, we calculate all n-grams at the word (rather than token) level.
3

---

Oper GPT2 1LT
Oper-GeT2 GPT2 343M
Port PyRlo JECM
Part Marbo 153M
Parc WWVY 1854
6.75
Heurisfe
6.23
I-grann Loga ooe
Poorsa Condaagon Cooffccinn
2-gram Loga vit
a 149
Pycre SUM
Pyd N9
Fybra 100M
Pye AUN
6.75
graan Loga vos
2.30
0.25
S-grni Eppride
2.00
(MB) Jiweeggee)
Pytra 1D
2
Pyea 0.88
Pyeh 120
Semeng: Simburty
Samere: Sieeluit
(C2 SGFTT
0.00
Figure 1: Pearson correlation coefficient between language model log-probability and heuristic metrics (n-gram log- bog-probability and word embedding cosine similarity). We show the mean values for all models across seeds and their 95% confidence intervals.
To calculate n-gram probabilities, we use infini-gram (Liu 2024). which allows user to retrieve the total counts of a given sequence in a corpus. For The Pile, we use the infini-gram API to access word counts; and for DeeeWe,,ext, we build our own infini- infini-gram index locally. We estimate n-gram probability based on these counts using Stupid Backoff as described by Brants et al. (2007). We provide the full code for building the index and calculating probabílities, and further details of our implementation in Appendix B
3.1.3 Contextual Similarity
To calculate contextual similarity, we follow previous work investigating the relationship between word embedding similarity and language model log-probability (Michaelov al. 2024). Specifically, we use fastText (Bojanowski et al. 2017: Grave et al. 2018) to calculate the word embeddings of the cosine
each critical word and the words in their context, and take contextual similarity similarity between the embedding of the critical word and the mean of the embeddings of all words in the context. We calculate this for both the fastText embeddings trained on Wikipedia (Bojanowski et al. (017)) and those trained on Common Crawl (Grave et al. (018). Additionally, for each of mean and weighted mean based on the SGPT approach (Muennighofi 2022). We provide further details of our implementation in Appendix
these, when calculating the context embedding. we calculate both the
3.1.4 Evaluation Dataset
We carry out our analysis on set of words in sentence contexts, which were all sampled from the FineWeb corpus (Penedo et al., (024). Specifically, we constructed set of words that were single tokens for all models, and that occurred as the fifth or later word (unique) sentences that began with capitalized word, had other capitalized words, were assigned probability of being toxic of 0.1 or less (using the model released by Logaccevva al. 2022), and, to avoid possible contamination, are not in the training data of any of the language models tested (based on infini-gram counts as described Section 1.2.2.. Our final dataset of Natural Words Context (NaWoCo) is made up of training set of 77, 77,999 items, validation set of 39,474 items, and test set of 40,980 items. Further details of dataset construction are provided in Appendix
3.1.5 Analysis
To investigate the relationship between the heuristics (n-gram log-probability and semantic similarity) and the language models, we use each model at each training checkpoint to calculate the log- probability of every word in our training set. We then calculate the correlation between the log- probabilities of every transformer at every time step and each of the heuristics.

---

3.2 Results and Discussion
We show the Pearson correlation between each heuristic and language model log-probability in Figure [ (for Spearman correlation, see Appendix ED. As can be seen, one pattern is remarkably consistent across models-over the course of training, we see peaks in the correlation between transformer log-probability and n--gram log-probability for increasing values of n. As in Chang et al. (2024), we see that language models become more sensitive to higher-order n-grams as they continue to train. Going beyond past work, we see that this observation applies not only to transformers, but to autoregressive language models of non-transten tonttransf architectures as well. We also see that the pattern observed by Chang et al. 20224) can be is explained in our data by both an increase in the correlation with the higher-order n- n-grams over the course of training and a simultaneous decrease in the correlation with the lower-order n n-grams- that is, the predictions move toward the former and farther away from the latter. In addition, as can also be seen in Figure [ larger models also see a greater decrease in the correlation to smaller n-grams, suggesting that their probability distributions shift farther from lower-order n-grams over the course of training. These last two findings align with previous work showing that the Kullback-Leibler divergence (Kullback and Leibler [195]) between Pythia models' output probability distributions and the probability distributions of unigram and bigram models show a similar pattern (Belrose et al. 2024).
Together with the idea that a greater number of parameters increases language models' capacity (see, e.g., Wang et al. 2017] Hestmess et al.] [017]] Kaplann et al.) [020 Allen-Zhu et al.) 2019 Allen-Zhu and Li 2024), these results may suggest that the smaller models may need to rely more on such lower-order ni-gra-llike predictions, while larger models may be able to learn more complex relations
/
between words as they continue to train.
We see two clear patterns with semantic similarity. First, there is little difference between the uniformly-weighhed and SGPT-weighted versions of each similarity metric, though the SGPT- weighted variant of the Wikipedia-basee fastText word vectors does appear to be consistently more correlated with language model log-pro pog-robbability than the uniformly-weighted variant. There does appear to be large difference between the two fastText versions, however. Specifically, vectors have a higher overall correlation to
see
that similarities derived from the Maamon-Crawl-aae log- probabiiiiy which peaks roughly the same time unigram log- probability does; while the Wikipedia-baie baased metrics have a lower correlation overall which peaks more concurrently with that of trigram log-probann nog-robabblity. This likely relates to the correlation between the similarity metrics and unigram log-probability- the Pearson correlation coefficient between the Common-Crant -aann--Cral--a-eeeee sim- ilarities and each metrie of unigram log-prohability is between 0.67 and 0.69, while the correlations with Wikipedia- similarities lie between 0.34 and 0.35 (see Appendix H
Finally, we note that the patterns across random seeds are remarkably similar-the confidence intervals are virtually invisible for the most part. Seed-level analyses (Appendix P) reveal that the larger confidence intervals in the first 10 steps of the Pythia models with PolyPythia seeds (i.e., the 14M-410M models) are due to small differences between models in the early stages of training; while the large confidence interval in the last steps of the Open- GPT2 345M models is driven by a single outlying checkpoint (step 256,000 of the beren (seed 49) model). In fact, Pathiay Parc-Mamba, and Parc- show Pearson correlation 2 0.93 overall at each step > 80; not only across seeds of the same model, but also across architectures (see Appendix
4 Experiment 2: Predicting Language Model Log-Probabilities with Heuristics
Experiment revealed patterns in the correlation between simple heuristics and language model log- pee-robbbbilit. But to what extent do these heuristics correspond to dissociable facets of language model behavior? On the one hand, all the heuristics are at least weakly correlated (see Appendix H presenting possible confound to interpretation. As an example, Common -aaon--Crrwleeeee -derived semantic similarity is highly correlated with unigram log bog--rbabbllity (r = 0.67 0.69 depending on weighting) and its correlation with language model log probaabiiyy follows the same trajectory. Thus, it is possible contextual semantic similarity and language
that the correlation between Common-Crawl-derre model log-probability may be largely explained by the former simply capturing unigram probability (i.e., frequency) for example, it is plausible that lower-frequency words will have lower-quality word embeddings and occur in fewer contexts, reducing their average similarity to other words. On
5

---

the other hand, this is clearly not the case for all heuristics we see different patterns in the degree of correlation between language model log- pog-prbbabillties over the course of training and different n-gram hog-probabilities, as well as unique pattern for Wikipedia-derived contextual semantic similarity.
Additionally, there are reasons to believe that language models may in fact learn make predictions that align with the n-gram and contextual similarity heuristics implicitly as part of autoregressive language modeling, based on the nature of the task itself. For example, word that is more common (i.e., that has a higher unigram probability) is by definition more likely to occur again, and similarly, a word that often follows a specific sequence of words (i.e.. that has a high n- n-gram probability for >> 1) is likely to do so again. In similar way, in a coherent text, one should generally expect a given word to be close in meaning (i.e., contextually semantically similar) to the words in its preceding context (for discussion, see Michaelov and Bergen [022] Michaelov et al.) 2023] (025). Indeed, a
sensitivity to contextual semantic similarity may at least partly explain how language models are
able to make predictions that align with real-world knowledge (as in, e.g.. /Zllers et al. [0018] 2019)
Forbes et al. 2019 Bisk et al. 2020 Sakaguchi et al.) [220]] [Jones et al. 2022] Kauf et al., 2023).
There is also mechanistic evidence of transformers directly implementing n-gram prediction (Bietti
cct al. 2023: Wang et al. 2025; Chang and Bergen, 2025;; and the fact that transformers directly
learn associations between tokens (e.g.. Meng et al. 2022 Bietti et al. 2023 Nichani et al.) 2024)
and appear to be able to make predictions at the concept as well as token level (Feucht et al. 2025)
suggests that prediction based on contextual semantic similarity is something that could in principle
be learned.
In this experiment, we focus on precise characterization of language model behavior in terms of how relates to the aforementioned heuristics. First, we revisit the finding in Section 3 that over the course of training, language model log-probabilities correlate better with n-gram log-probabilities of increasing order, and correspondingly correlate worse with the log-probabilities of lower-order n-grams. We ask whether language model predictions are still biased toward lower-order n-gram probabilities even as they begin to reflect higher-order n-grams, Specifically, we test whether language model log-probability is sensitive to unigram log-probability above and beyond the extent to which this is captured in 5- 5-gram probabiitb... Second, we ask whether language model predictions are sensitive contextual semantic similarity above and beyond that implicitly captured by unigram and 5 5-gram probability. This is crucial because, as previously stated, unigram probability may impact contextual semantic similarity metrics due to its effect on the word vectors learned; and, as stated, coherent sentences are likely to involve words that are semantically similar to their context, and thus, if these sequences are learned, they may implicitly contain contextual semantic similarity information.
As far as we are aware, these questions are novel in language model research. But they have been studied in the context of human language processing. Specifically, number of studies have investigated whether there are dissociable effects of word frequency, contextual probability (derived from n-grams neural language models), and semantic similarity on neural and behavioral indices of prediction in humans, with varying results (Lau et al.) [2013] Frank and Willems 2017, Nicuwland eet al. 2020 Shain 2024 Michaelov et al. 2024 Opedal al. 2024). We follow the approach taken in these studies, using the n-gram log-probabilitte and similarity metrics to predict language model oog-probability with linear regression, allowing us estimate the extent to which language models show a bias in prediction toward each heuristic while controlling for the possible influence of the others.
4.1 Method and Analysis
The evaluation dataset, cosine similarities, and n-gram and language model log-probabilities were all the same as in Experiment 1. Instead of computing raw correlations, we instead fit lincar regressions that predict the dependent variable-in this case, language model log-probabilit- based on all predictors of interest, namely, unigram log-probability, 5-gram tog-probability, and contextual semantic similarity. Thus, as in human studies, we are able to investigate whether correlations between unigram language model log probability and our predictor variables are dissociable.
We are also interested in the robustness of these patterns. Thus, in addition to using only n-grams calculated from the same corpus the language model was trained on (i.e., either OpenWebText or The Pile), we also carry out the same analysis with the n-grams based on the other corpus. For the previously-diseusee
reasons, we are also interested whether there is a difference between the two similarity metrics, and we also construct regressions with each. Thus, all regressions

---

[A] Regression confiicients
Oper GPT2 LL7N
Open GPT2 34GM
Forc Pyffia 1869
Parc: Mambo 133M
Par: HWXY J83M
0.15
Heuristic
experite pilertdd
0.2
Mirnaot vell
ayca
Pytha IIM
Pysa
Pytra ZOOM
Pytha -ION
2-gore Losprato
0.71
5-gare Logeribs
0.50
0.23
plabted n-gramm)
Sera Siss arly
Fyraa
Pytro
Fytri
1.00
Similerty Metric
Weightd)
0.50
Semmatte Sim wnly
0.00
10 10 Training Stace
explained heuristics
Open-6.177 17M
Open-GPT) 3410
Pam--lyth 1920
Part-laamio 132M
Fart-RWcy 16AM
0.50
0.2
Splt
1.00
1.. Validation Set
Traniig
Wild aimdurty
0.00
Mascnod pram
Pyth
Pytee
Pytra
Pytaa
Pyte
pegrem cibarryy
1.00
0.55
CC seniarty
E-gran-
0.50
Training
Figure 2: (A) Regression coefficients of the three heuristics over the course of training under different conditions, specifically, whether the r-gram data the same as that on which the language model was trained (matched) not (unmatched), and whether SGPT-weighted contextual semantic similarity metric is calculated using Common-Cras baseed or Wikipedia based fastText word vectors. (B) Proportion of the variance language model log-probability explained by the regressions in Figure 2
We also report the R² values of the same regressions' predictions on the validation set.
include 3 predictors: unigram log probabiblity (OpenWeb'Tes) The Pile), 5-gram log-probability (ee'eWeeTexxt or The Pile; the same corpus as unigram log- bog-probability), and semantic similarity (Wikipedia-bart based Cemmon-Craal-bassee
In order to easily compare the coefficients, z-transform all variables in the regression (we provide the coefficients for the same regressions with un-normalized variables in Appendix K
4.2 Results and Discussion
We present our results for all regressions with SGPT-weighted contextual semantic similarity in Figure 2] We find that as in Experiment 1, while the exact steps which the phases themselves occur differs somewhat across models, the phases themselves are remarkably clear and consistent.
In Phase 1, the coefficient of unigram log probbabbyyyltyy on language model log probability increases sharply from zero (or near-zero). This increase peaks the same time as the peak Pearson correlation seen in Figure I Concurrently, the coefficient of semantic similarity (both metrics) increases. By contrast, there no positive increase the coefficient of the 5-gram fog-probability- in fact, it slightly decreases below zero during this phase.

---

In Phase 2. the coefficient of unigram log probaabiiyy decreases and the coefficient of 5-gram probability increases. There is also sharp increase in the coefficient of 5-gram log- proobabbiillty.. At or soon after the beginning of this phase is small trough in the coefficient of dikipedia-derived similarity, which is smaller (or nonexistent) for
similarity.
In Phase 3, the change in coefficients slows down and begins to stabilize for all models.
These patterns are consistent across all models, with the main difference being in the timing of the phases. In the Pythia models, phases generally occur earlier in the larger models than in the smaller models, but in the two Open-GPT) models, this pattern is reversed. It notably does not appear that training tokens explain the timings of the phases better than training steps- in fact, despite being trained Pare-Pythia, Parc-RWKY. V, and Parc-Mamba) generally enter phases in fewer steps than the Pythia models which are trained coefficient size-smaller models see a smaller drop in the coefficient of unigram log- bog-rrbbability and smiler increase in the coefficient of 5-gram log-probability than larger models. There is also some variability in the peaks and troughs of the coefficients of the semantic similarity metrics, though they all appear follow the general trajectory described above. To further verify the robustness of these patterns, we also look at the seed-level results for regressions including both SGPT-weighted and unweighted variants of the similarity metrics (provided in Appendix D. which produce virtually identical patterns of coefficients.
0.5M tokens per step, the models trained on Open Text (Open-GPT2,
2M tokens per step. Model size, however, does appear to impact
We next turn to the overall fit of the regressions language model log-probability. The R2 of each regression provides an estimate of the proportion of the variance in language model log- probability explained by the regression. Overall fit is largely shaped by the strength of the effect of unigram log- peg-probability-tee
regressions best predict language model log-probability (1t² 0.86 0.98 depending on the model) when the effect of unigram log-probability on language model log-probability is at its peak, after which fit decreases sharply as the strength of the effect of unigram log-probabie probabbbbiy decreases sharply, and levels off as the decrease in the strength of unigram log- bog-robbaiility slows down. However, the overall fit still remains relatively high even after this decrease- it stabilizes for the models with fewer than 0.5B parameters and does not fall below R = 0.5, even for the largest models. In fact, except for the carliest steps of training (Pythia 14M: step <128; Pythia 31M: step <64; all other models: step <32), the heuristics always explain at least half of the variance in model log-probability. As can be seen, the choice of n-gram corpus and fastText training corpus do not dramatically change any of these patterns, which are also relatively consistent across seeds (see Appendix D). This suggests that overall fit is robust to these factors. To further evaluate the robustness of our regressions to overfitting, we calculate the I22 on the held-out validation set using predictions from the regressions. Again, we see almost no difference, suggesting that these results are robust. We again see virtually identical patterns when we look at seed-level results for both the SGiT-weeghted and unweighted similarity variants (sce Appendix(] D).
5 General Discussion
In line with the results of previous work (Karpathy et al., 2016 Choshen et al., 2022: Chang and Bergen [022] Chang et al.) 2024)) we show in Experiment that the extent to which language model predictions correlate with n- n-gram predictions varies over time such that language models trained on more data make predictions that correlate more strongly with higher-order regrams. We show that for n E {1, 2, 3, 4, 5), this applies to transformers across 3 orders of magnitude (1 (14M to 12B parameters), as well as language models of both the RWKV and Mamba architecture. We also find predictable impact of model scale: while smaller models eventually make predictions that are as correlated (or almost as correlated) with higher-order n n-grams the larger models, they retain higher correlation with the lower-order ni-rams than the larger models. As the larger models' predictions increasingly correlate with the higher-order n-grams, their correlation with the lower-order n-grams decreases. This suggests that the larger models shift to predicting in line with higher-order n-grams, while the predictions of the smaller models stay closer to those of the lower-order n-grams.
Experiment 2 confirmed this through multiple linear ungres log-probability maintains a larger coefficient during the later phases in smaller than larger models, while the reverse is true of 5-gram log-prod bog-robbbbility. This means that higher-order n-grams account more for the behavior of larger models than they do for smaller models. This could be taken to suggest that the smaller models
8

---

may rely on lower-order n g-gram relations perhaps due to limited capacity- while larger models may be able to more effectively make use of longer contexts. Additionally, the fact that the overall fit of the heuristics decreases the most for the largest Pythia models in the last phase of pretraining combined with the fact that they already begin to perform better than smaller models at standard benchmarks during this phase (Appendix D suggests that the predictions of these models are likely to reflect other more complex cues at this stage of training. Thus, our results seem to suggest that while language model predictions will (and perhaps must) show a high degree of correlation with the heuristics during training. it is only once they pass this stage that that they begin to perform well on downstream tasks. Indeed, while the exact relationship between the two is not known, research has shown that in-context n 2-gram heads only occur in a model after it has learned to predict n- n-grams in the training data (Bietti al. 2023 Wang et al. 2025). ). Crucially, in-context n-gram heads such as induction heads are thought to greatly contribute to language models' in-context learning capabilities
Bietti et 2023 Akyürek al.. 2024 Edelman et al. 2024 Chen et al., 2024b) Crosbie and Shutova 2025). which in turn have been argued to be responsible for wide range of language model behaviors, including performance on downstream tasks (see, e.g.. Lampinen et al. (025). Despite this, there is also evidence that the tendency for language models to make predictions that align with these heuristics persists throughout the course of training. In fact, has been shown that as language models get larger and are trained on more data, they continue to (and may even increase the extent to which they do) predict eigh-probbbility n-grams (McKenzie et al.,| |023] Michaelov and Bergen 2023) and words that are semantically related to their context (Michaelov et [025| Gonen et al., 2025), even when they should not. An interesting line of future work would be to investigate whether it is possible to predict the extent to which language models are susceptible to these kinds of phenomena based on analyses such as those carried out in Experiment 2.
(Elhage et al. [2021] Olsson et al. 2022
Are the predictions of larger models trained on more data biased toward lower-order n-- n-grams above and beyond the extent to which they are accounted for by higher-order n-grams? On the one hand, decreases in the size of the I-gram coefficient are usually matched by increases in the size of the 5-gram coefficient, and vice-versa; suggesting some degree of competition or incompatibility between the two. On the other hand, even as they become increasingly sensitive to higher- higher-order n n-gram probabilities. lower-order n-gram probabílities (or at least, unigram probabilities) are still overweighted in language model predictions: that is, the models over wer-ppedicct words with high unigram probability relative their 5-gram probability. This is in line with work showing that lower-order n-gram crcuuts--ppccfficllly bigram circuits- persist over the course of training in transformer models, though they appear to become less effective and less distinct in later stages (Chang and Bergen 2025). Whether this is also true for the RWKV and Mamba models is a question for future work.
Previous work has demonstrated a relationship between the probabilities assigned by language models to words in context and the semantic similarity between those words and their contexts (Michaelov) let al., | 2024). In this paper, we provide what is, as far as we are aware, the first investigation of both the extent to which this correlates with predictions above and beyond other factors thought to impact them (in this case, n- n-gram probability). and the extent to which this varies over the course of training. We find that while the direct correlation between semantic similarity and language model log- prg-obbabiyltyy varies depending on how semantic similarity is calculated, after accounting for unigram and 5- 5-gram prg--rbbability, most of this difference disappears we see the correlation with similarity emerge relatively early, and remain throughout training.
Additionally, we find that while the weighting method used create the context vector-uniform vs. SGGTTwwighting--o0
not appear to have much of an impact, there was a difference between the two different sets of vectors we used. Specifically. the Common-Crawe Caatoon--rawl-bseeed fastText vectors (Grave et al. [2018] 8) are more closely correlated with unigram probability than the Wikipedia- basee based vectors (Bojanowski et al. 2017), , which could explain the differences we see in their relationship to language model bog-robabiliit. What explains the difference between these embedding vectors? One intuitive possibility is that the difference is due to differences in training corpus. However, another plausible explanation is that it relates to how the embeddings were trained. Specifically, the Wikipedia embeddings were trained using a skip-gran
model (Bojanowski et al., (017)) and the Common Crawl vectors were trained using a continuous bag-of-words (CBOW) model (Grave et al.) 2018). Previous work has suggested that skip-gram models learn higher-quality representations for infrequent words (Mikolov et al. 2013), and thus, a likely reason for the higher correlation between
9

---

the CBOW-based embeddings and unigram frequeney is simply that lower-frequent words have lower-quality representations, and thus their similarity to the context under-estimated.
Our results have several key implications. First, they provide evidence that across architectures and scales, there is consistent relationship between the predictions of language models and n-gram probability. A strong interpretation of our results, combined with similar results on recurrent neural networks by Karpathy et (2016), is that system engaged in autoregressive language modeling will inevitably go through phases of overfitting to n-grams of increasing 7t before learning more abstract patterns. We also for the first time observe a consistent correlation between semantic similarity and language model predictions over the course of training, something that can be detected even after accounting for any implicit semantic similarity relations inherent n-grams (i.e., based on co-occurrence). Finally, while it has previously been demonstrated that it is possible to construct rule based on n-gram statistics that can explain a given language model prediction (Nguyen 2024), we see that, except for in the very early stages of training, there is specific weighting of three simple heuristics (plus constant) that can explain over half the variance in the predictions made by any seed of any language model tested at any point during training.
6 Limitations
While our results are consistent across our models, our analysis limited three architectures, and in the case of Mamba and RWKV, only relatively small models (130-169M models trained on ~2B tokens). Thus, is possible that other models could show different training dynamics. We also limit our analysis to n-grams in E {1. 2, 3, 4, 5} and static word embeddings, but language models may be sensitive to higher-order n-grams and semantic similarity that is better reflected by contextual word embeddings or text embeddings. Finally, our regressions still do not account for all the variance language model behavior- still a lot to understood, even in the smallest models.
Conclusions
Language model behavior throughout training can largely be accounted for by several simple heuris- ties: word frequency, n-gram probability, and semantic similarity. The extent to which each of these correlates with language model predictions has a consistent pattern across models of different sizes, models trained on different data, and models with different architectures. Taken together, this suggests that the autoregressive language modeling task itself may be the largest factor-an perhaps the decisive one-in shaping the behavioral phases that language models pass through. It may be that models cannot help but first crawl through n-gram predictions in their gradient descent toward mature next-word prediction.
Acknowledgments
We would like to thank the members of the Computational Psycholinguistics Laboratory at MIT and the Language and Cognition Laboratory at UCSD for their valuable advice and discussion. James Michaelov was supported by a grant from the Andrew W. Mellon foundation (#2210-13947) during the writing of this paper.
References
Akyürek, E., Wang, B., Kim, Y., and Andreas, J. (2024). In-Context Language Learning: Architectures and Algorithms. In Proceedings of the 41st International Conference on Machine Learning, pages 787-812. PMLR. ISSN: 2640-3498.
Allen-Zhu, Z. and Li, Y. (2024). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. arXiv:2404.05405 [cs].
Allen-Zhu, Z., Li, Y., and Song, Z. (2019). A Convergence Theory for Deep Learning via Over- Parameterization. In Proceedings of the 36th International Conference on Machine Learning, pages 242-252. PMLR. ISSN: 2640-3498.
Anderson, P. W. (1972). More Is Different. Science, 177(4047)
10

---

Andonian, A., Biderman, S., Black, S., Gali, P., Gao, L., Hallahan, E., Levy-Kramer, J., Leahy, C.. Nestler, L., Parker, K., Pieler, M., Purohit, S., Songz, T., Wang. P., and Wein- bach, S. (2021). GPT-NeoX: Large scale autoregressive language modeling in PyTorch. https://zenod..orgg/ooo http:////wddoooorrr////////////1
Baker, J. (1975). The DRAGON system-An An overview. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1):24-29,
Belrose, N., Pope, Q.. Quirke, L., Mallen, A., and Fern, X. (2024), Neural networks learn statistics of increasing complexity. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of ICML '2, pages 3382-3409, Vienna, Austria. JMLR.org.
Biderman, S., Prashanth, U., Sutawika, I.., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. (2023a). Emergent and Predictable Memorization in Large Language Models. Advances in Neural Information Processing Systems, 36:28072-28090.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Wal, O. D. (2023b). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Proceedings of the 40th International Conference on Machine Learning, pages 2397-2430. PMLR. ISSN: 2640-3498.
Biderman, S., Schoelkopf, H., Sutawika, L.., Gao, L.,, Tow, J., Abbasi, i, B., Aji, A. F., Ammanamanchi, S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Hsu, Jaiswal, M. Lee, W. Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J., Skowron, A., Tan, S., Tang, X., Wang, K. A., Winata, G. I., Yvon, and Zou, A. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv:2405. 14782 [cs].
Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H., and Bottou, L. (2023). Birth of Transformer: A Memory Viewpoint. In Advances in Neural Information Processing Systems, volume 36, pages 1560-1588.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. (2020). PIQA:
Reasoning about Physical Commonsense in Natural Language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7439 Number: 05.
Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. (2021). GPT-Neo: Large scale autoregres- sive language modeling with mesh-tensorflow. https://doi.orr//100 http:///0002/1///oooo222200
Bojanowski, P., Grave, E.,, Joulin, A., and Mikolov, T. (2017). Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135-146. Place: Cambridge, MA Publisher: MIT Press.
Brants, T., Popat, A. C., Xu, P., Och, J.J., and Dean, J. (2007). Large Language Models in Machine Translation. In Eisner, J., editor, Proceedings of the 2007 Joint Conference Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP- (EMLLL.CONLL), pages 858 867, Prague, Czech Republic. Association for Computational Linguistics.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, L., and Amodei, D. (2020). Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1807-1901. Curran Associates, Inc.
Chang, T. A. and Bergen, B. K. (2022). Word Acquisition in Neural Language Models. Transactions of the Association for Computational Linguistics, 10:1-16.
Chang, T. A. and Bergen, B. K. (2024). Language Model Behavior: A Comprchensive Survey. Computational Linguistics, 50(1):293-350.
Chang, T. A. and Bergen, B. K. (2025). Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models. arXiv:2504.1547 [cs].

---

Chang. T. A., Tu, Z., and Bergen, B. K. (2024). Characterizing Learning Curves During Language Model Pre-Tree rre-riainng: Learning, Forgetting, and Stability. Transactions of the Association for Computational Linguistics, 12:1346-1366 Place: Cambridge, MA Publisher: MIT Press.
Chen, A., Shwartz-ZDi -ivi, R., Cho, K., Leavitt, M. L., and Saphra, N. (2024a). Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs. In The Twelfth International Conference Learning Representations.
Chen, S., Sheen, H., Wang, T., and Yang. Z. (2024b). Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Choshen, L., Hacohen, G., Weinshall, D., and Abend, O. (2022). The Grammar-Learinng Trajectories of Neural Language Models. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8281-8297, Dublin, Ireland. Association for Computational Linguistics.
Chung, H. W., Garcia, X., Roberts, A., Tay, Y., Firat, O., Narang, S., and Constant, N. (2022). UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining. In The Eleventh International Conference on Learning Representations.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. arXiv:1803.0545 [es].
Crosbie, J. and Shutova, E. (2025). Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning. In Chiruzzo, L., Ritter, A., and Wang, L.,, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 5034-5096, Albuquerque, New Mexico. Association for Computational Linguistics.
Du, Z., Zeng, A., Dong, Y., and Tang, (2024). Understanding Emergent Abilities of Language Models from the Loss Perspective. Advances in Neural Information Processing Systems, 37:53138- 53167.
Edelman, E., Tsilivis, N., Edelman, B. L.., Malach, E., and Goel, S. (2024). The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains. Advances in Neural Information Processing Systems, 37:64273-64311
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A.. Conerly, T., DasSarma, N., Drain, D., Ganguli, D.,, Hatfield-Dodds, Z.., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread.
Feucht, S., Todd, E., Wallace, B. C., and Bau, D. (2025). The Dual-Route Model of Induction. In Second Conference on Language Modeling.
Forbes. M., Holtzman, A., and Choi, Y. (2019). Do Neural Language Representations Learn Physical Commonsense? The 41st Annual Meeting of the Cognitive Science Society, Montreal, Quebee, Canada.
Frank, S. I.. and Willems, R. M. (2017). Word predictability and semantic similarity show distinct patterns of brain activity during language comprehension. Language, Cognition and Neuroscience. 32(9):1122-1203
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs].
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Mc- Donell, K., Muennighoff, N., Phang, J., Reynolds, I.., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2021). framework for few-shot language model evaluation. http:///bb.ooo/00.222222ooo000..
12

---

Gokaslan, A. and Cohen, V. (2019).
OpenWebText
corpus.
http://Skylion007, http:///kkloo0000ggiimbooooeeee wttphbbbiio/ppeeeeeeeeooooo
Gonen, H.. Blevins, T., Liu, A., Zettlemoyer, L., and Smith, N. A. (2025). Does Liking Yellow Imply Driving School Bus? Semantic Leakage Language Models. Chiruzzo, L.,, Ritter, A., and Wang, I.., editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages 785-798, Albuquerque, New Mexico. Association for Computational Linguistics.
Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, (2018). Learning Word Vectors for 157 Languages. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Gu, A. and Dao, T. (2024). Mamba: Limear-Time Sequence Modeling with Selective State Spaces. In First Conference on Language Modeling.
Hawkins, D. (2023). arwen is checkpoint progression outlier? cffm//msstrlliisuus/199
https//ggihub.com/ssnffttt
Hestmess, J., Narang. S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang. Y., and Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv: 17Xiv:1712.00409 [cs].
Hu, J. and Frank, M. (2024). Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling.
Jelinek, F., Bahl, L., and Mercer, R. (1975). Design of linguistic statistical decoder for the recognition of continuous speech. IEEE Transactions on Information Theory, 2103):250-266. Conference Name: IEEE Transactions Information Theory.
Jones, C. R., Chang, T. A., Coulson, S., Michaelov, J. A., Trott, S., and Bergen, (2022). Distrubutional Semantics Still Can't Account for Affordances. Proceedings of the Annual Meeting of the Cognitive Science Society, 44(44).
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.0883 [es].
Karamcheti, S., Orr, L.,, Bolton, J., Zhang, Goel, K., Narayan, A., Bommasani, R., Narayanan, D., Hashimoto, T., Jurafsky, D., and others (2021). Mistral journey towards repro- ducible language model training. Stanford Center for Research on Foundation Models: https://crfm.stannn wtw.sood.dddun20000. http://ddddd/e/22/////////// /00/////staa/ html.
Karpathy, A., Johnson, J., and Fei-Fei, L. (2016). Visualizing and Understanding Recurrent Networks. In International Conference on Learning Representations (Workshop Track).
Kassner, N. and Schütze, H. (2020). Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. In Proceedings of the 58th Annual Meeting of the Association for Computaational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fe- dorenko, E., and Lenci, A. (2023). Event Knowledge Large Language Models: The Gap Between the Impossible and the Unlikely. Cognitive Science, 47(1 (7(11):e13366 _eprint: www.://rrrraaiillll.oooo////p/po 10.o//////.1111///133333336
Kneser, R. and Ney, H. (1995). Improved backing-off for M-gram language modeling. In /995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 181-184 1. ISSN: 1520-6149.
Kullback, S. and Leibler, R. A. (1951). Information and Sufficiency. The Annals of Matematical Statistics, 22(1):79-86 -6. Publisher: Institute of Mathematical Statistics.
13

---

Lampinen, A. K., Chan, S. C. Y. Singh, A. K., and Shanahan, M. (2025). The broader spectrum of in-context learning. arXiv:2412.03782 [es].
Lau, E. F., Holcomb, P. J., and Kuperberg, G. R. (2013). Dissociating N400 Effects of Prediction from Association in Single-word Contexts. Journal of Cognitive Neuroscience, 25(3):484-500 502.
Laughlin, R. B. and Pines, D. (2000). The Theory of Everything. Proceedings of the National Academy of Sciences, 97(1):28-3) 31. Publisher: Proceedings of the National Academy of Sciences.
Liang, L. (2024). Benchmark tests giving different results. seluuaaion-harnssssssssss//
https//gittub.comFEeeeeeee/////
Linzen, T., Dupoux, E., and Goldberg, Y. (2016). Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Liu, J., Min, S., Zettlemoyer, L.., Choi, Y., and Hajishirzi, H. (2024). Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens. In First Conference on Language Modeling.
Logacheva, V., Dementieva, D., Ustyantsev. S., Moskovskiy, D., Dale, D., Krotova, I., Semenov, N., and Panchenko, A. (2022). ParaDetox: Detoxification with Parallel Data. Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6818-6818, Dublin, Ireland. Association for Computational Linguistics.
Magnusson, I., Bhagia, A., Hofmann, V., Soldaini, L.,, Jha, A. H., Tafjord, O., Schwenk, D., Walsh, F. P., Elazar, Y., Lo, K., Groeneveld, D., Beltagy, I., Hajishirzi, H., Smith, N. A., Richardson, K., and Dodge, J. (2024). Paloma: A Benchmark for Evaluating Language Model Fit. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Marcus, M. P., Santorini, B., Marcinkiewicz, M. A., and Taylor, A. (1999). Treebank-3. https//catalog.. http://ccaalog..d..uumee www.uuenn.ddwLLDC999...
Markov, A. A. (1913). Essai d' unu recherche statistique sur texte du roman "Eugene Onegin" illustrant la liaison des epreuve en chain ("Example of a statistical investiga- tion of the text of "Eugene Onegin" illustrating the de- pendence between samples in chain'), Izviestiia Imperatorskoi akademii nauk (Bulletin de I' l'Académie impériale des sciences de St.-Pétersbourg), 7:153-162.
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D.,, and Griffiths, T. L. (2024). Embers of autore- gression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, (21(41):2222222012
McKenzie, I. R., Lyzhov, A., Pieler, M. M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Shen, X., Cavanagh, J., Gritsevskiy, A. G., Kauffman, D., Kirtland, A. T., Zhou, Z.. Zhang, Y., Huang, S., Wurgaft, D., Weiss, M., Ross, A., Recchia, G., Liu, A., Liu, J., Tseng, T., Korbak, T., Kim, N., Bowman, S. R., and Perez, E. (2023). Inverse Scaling: When Bigger Isn't Better. Transactions on Machine Learning Research.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 15:17359-17372
Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2017). Pointer Sentinel Mixture Models. In International Conference Learning Representations.
Michaelov, J. A., Bardolph, M. D., Coulson, S., and Bergen, B. K. (2021). Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude? In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society, pages 300-306, University of Vienna, Vienna, Austria (Hybrid).
Michaelov, J. A., Bardolph, M. D., Van Petten, C. K., Bergen, B. K., and Coulson, S. (2024). Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects. Neurobiology of Language, 1(1):107-135.
14

---

Michaelov, J. A. and Bergen, B. K. (2022). Collateral facilitation in humans and language models. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 13-26, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Michaelov, A. and Bergen, of Pretraining. In Bouamor, H., Pino. J., and Bali, K., editors. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14607-14615, Singapore. Association for Computational Linguistics.
(2023). Emergent Inabilities? Inverse Scaling Over the Course
Michaelov, J. A., Coulson, S., and Bergen, B. (2023). Can Peanuts Fall in Love with Distributional Semantics? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45.
Michaelov, J. A., Estacio, R., Zhang. Z., and Bergen, B. (2025). Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events. In Che, W., Nabende, J., Shutova, E.,, and Pilehvar, M. TT,, editors, Findings of the Association for Computational Linguistics: ACI. 2025, pages 15528-13551. Vienna, Austria. Association for Computational Linguistics.
Mikolov, T., Lc, Q. V., and Sutskever, I. (2013). Exploiting Similarities among Languages for Machine Translation. arXiv:1309.4411 [cs].
Misra, K., Ettinger, A., and Rayz, J. (2020). Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming. In Findings the Association for Computational Linguistics: EMNLP 2020, pages 4625-4635, Online. Association for Computational Linguistics.
Muennighoff, N. (2022). SGPT: GPT Sentence Embeddings for Semantic Search. arXiv:2202.0890 [cs].
Nguyen, T. (2024). Understanding Transformers via N-Gram Statistics. In Advances in Neural Information Processing Systems, volume 37, pages 98049-98088
Nichani, E., Lee, J. D., and Bietti, A. (2024). Understanding Factual Recall in Transformers via Associative Memories. In The Thirfeenth International Conference Learning Representations.
Nicolis, G. and Prigogine, I. (1977). Self-organization in nonequilibrium systems: from dissipative structures order through fluctuations. Wiley, New York.
Nieuwland, M. S., Barr, D. J., Bartolozzi, F., Busch-Moreno, S., Darley, E., Donaldson, D. I., Ferguson, H. J., Fu, X., Heyselaar, E., Huettig, F., Matthew Husband, E., Ito, A., Kazanina, N., Kogan, V., Kohút, Z.., Kulakova, E., Mézière, D., Politzer-Ahles, S., Rousselet, G., Rueschemeyer, S.-A., Segaert, K., Tuomainen, J., and Von Grebmer Zu Wolfsthurn, S. (2020). Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials. Philosophical Transactions of the Royal Society B: Biological Sciences, (75(1791)2018805)) Publisher: Royal Society.
OLMo Team, Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D.,, Brahman, F,, Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J., Murray, T., Nam, C., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer, L., Farhadi, A., Smith, N. A., and Hajishirzi, H. (2025). 2 OLMo Furious. arXiv:2501.0000 [cs].
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. (2022). In-context Learning and Induction Heads. arXiv:2209.11880 [cs].
Opedal, A., Chodroff, E., Cotterell, R., and Wilcox, E. (2024), On the Role of Context Reading Time Prediction. In Al-Onaizan, Y., Bansal, M., and Chen, Y.N.,, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3042-3058, Miami, Florida, USA. Association for Computational Linguisties.
15

---

O'Connor, T. (2021). Emergent Properties. In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, winter 2021 edition.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q.. Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pages 1525-1534, Berlin, Germany, Association for Computational Linguisties.
Penedo, G., Kydlfčkk, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. (2024). The FineWeh Datasets: Decanting the Web for the Finest Text Data at Scale.
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J. (2023). The WefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only. Advances in Neural Information Processing Systems, 36:79155-79172.
Peng, B., Alcaide, E., Anthony, Q.. Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derezynski, L.., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wind, J., Wožniak, S., Zhang, Z.,, Zhou, Q., Zhu, J., and Zhu, R.-J (2023). RWKV: Reinventing RNNs for the Transformer Era. In Bouamor, H., Pino, J. and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14048-14077, Singapore. Association for Computational Linguistics.
Pennington, J.. Socher, R., and Manning, C. (2014). GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners
Raffel, C.,, Shazeer. N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. (2020). WinoGrande: An Adversarial Wino- grad Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(5):8732-8740) Number: 05.
Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are Emergent Abilities of Large Language Models Mirage? In Advances in Neural Information Processing Systems, volume 36, pages 55565-55581. Curran Associates, Inc.
Shain, C. (2024). Word Frequency and Predictability Dissociate in Naturalistic Reading. Open Mind, 8:177-201.
Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3):379 423.. Conference Name: The Bell System Technical Journal.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, E., Zettlemoyer, L., Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. (2024). Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725-15788, Bangkok, Thailand. Association for Computational Linguistics.
Sun, X. and Lu, W. (2022). Implicit n-grams Induced by Recurrence. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1624-1639, Seattle, United States. Association for Computational Linguistics.
16

---

van der Wal, O., Lesci, P. filler-Eberstein, M., Saphra. N., Schoelkopf, H., Zuidema, W., and Biderman, S. (2024). PolyPythias: Stability and Outliers across Fifty Language Model Pre- Tre-Trainnng Runs. In The Thirteenth International Conference Learning Representations.
Voita, E., Ferrando, J., and Nalmpantis, C. (2024). Neurons in Large Language Models: Dead, N- gram, Positional. In Ku, L.-W., Martins, A., and Srikumar, V., editors. Findings of the Association for Computational Linguistics: ACL 2024, pages 1288-1301 Bangkok, Thailand. Association for Computational Linguistics.
Wang, M., Yu, R., E, W., and Wu, L. (2025). How Transformers Get Rich: Approximation and Dynamics Analysis. arXiv:2410.11474 [es].
Wang, Y.-X., Ramanan, D., and Hebert, M. (2017). Growing Brain: Fine-Tuning by Increasing Model Capacity. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), ISSN: 1063-6919.
pages
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang. S.-F, and Bowman, S. (2020). BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Weber, M., Fu, D. Y., Anthony, Q.. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. (2024). RedPajama: an Open Dataset for Training Large Language Models. Advances in Neural Information Processing Systems, 37:1 6466--114992
Wei, J.. Garrette, D., Linzen, and Pavlick, (2021). Frequency Effects on Syntactic Rule Learning in Transformers. In Moens, M.-F,, Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932-948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wei, J., Tay, Y., Bommasani, R., Raffel, C.,, Zoph, B., Borgeaud, S., Yogatama, D.,, Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang. P., Dean, J., and Fedus, W. (2022a). Emergent Abilities of Large Language Models. Transactions Machine Learning Research.
Wei. J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022b). Chain-of- of Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc.
WelbI, J.. Liu, N. F., and Gardner, M. (2017). Crowdsourcing Multiple Choice Science Questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94-106, Copenhagen, Denmark. Association for Computational Linguistics.
Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguisties.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguisties.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T.. Ott, M., Shleifer. S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. (2022). OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01 A00:2205500106 [cs].
17

---

A Language Model Details
A.1 Pretrained Models
We summarize the details of all models analyzed in Table Further details of each set of models are discussed in the relevant sections below. scc
Table I: Details of the models used in our analyses.
Model
Parameters
Seeds
Tok/Step
Steps
14M
10
31M
10
70M
10
160M
10
410M
10
1, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, 143000
Pythia
410M
2M
IB
1.4B
2.8B
6.9B
12B
117M
4
0, 10, 20, 40, 80, 100, 200, 400, 800, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, 256000, 400000
Open-GPTH
0.5M
345M
4
Parc-Py Pare-Pythia 160M
10. 20, 40, 80, 160, 320, 640, 1280, 2560, 4000
Pare-RWKV
169M
0.5M
Parc-Mamba
130M
Pythia The Pythia (Biderman et al. 2023b) models based on the GPT-Neo models (Black et al. [202]] Andonian et al.
of autoregressive language models
2021] ) which were created as an
attempt train fully-open versions of the smaller GPT-3 language models (Brown et al.) 2020). Thus, all models are trained on a 300B-token dataset (The Pile: Gao et al.
2020). Our analysis includes the
main (seed 1234) 14M, 31M, 70M, 160M, 410M, IB, 1.4B, 2.8B, 6.9B, and 12B parameter models. In addition, we use the recently-release
14M, 31M, 70M, 160M, 410M PolyPythia models (van der Wal al. 2024) trained further random seeds each (1, 3, 4, 6, 7, 9); thus, we have 10 models for each of these parameter sizes. With all Pythia models, we calculated word probabilities with model checkpoints the following steps: 1, 2, 8, 16, 32, 64, 128. 256, 512, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, and 143000 tfully-trained). Note that models were trained on 2M tokens at each step. Because of a known issue with the fp16 precision used during training (see Liang 2024), we run all models at fp32 precision.
Open-GPT2 We also carry out our analyses on the GPT-2 models trained as part of the Mistral project (Karamcheti et al., (021). These are set of GPT-2 small (117M parameters) and GPT-2 medium (345M parameters) language models trained on Open WebText (OWT) corpus (Gokaslan) and Cohen 2019), with training checkpoints released. For each size, models trained with 5 random seeds were released. Due to known issue with the GPT-2 Medium model with random seed (see Hawkins 2023), used the 4 remaining random seeds of each model (namely, seeds 49, 81, 343, and 777). For each of the models trained with each of these random seeds, we calculated language model probabilities at checkpoints corresponding to the following steps: 0, 10, 20, 40, 80, 100. 200, 400, 800, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, 256000, and 400000 (fully trained). Note that models were trained on 0.5M tokens at each step.
A.2 Pare Models
In order to investigate the effect of architecture, we also train our own models. Specifically, we train Pythia (Biderman et al. 2023b). Mamba-l (Gu and Dao 2024), RWKV-4 (Peng et al. 2023) models.
18

---

We selected one model of each architecture of comparable size namely, Pythia 160M, RWKV 169M, and Mamba 130M. These models were adjusted these models to all use the Pythia tokenizer, and then trained from scratch (i.e., random initialization) on the Open Web corpus for 4000 steps, where each step consisted of 1024-token length sequence with batch size of 512, for total of 0.5M tokens, following Karamcheti et al. (202T). All hyperparameters were chosen to match those of the original model training where possible (Biderman et al., 2023b) Peng et al. 2023 Gu and Dao 2024). We provide the full training code.
For each of the three models selected, we trained six models with different random seeds (0, 1, 2, 3, 4, 5), each of which seed was applied both to the data collator and initialization. We analyze data from steps 10, 20, 40, 80, 160, 320, 640, 1280, 2560, and 4000 (fully-trainee
B Estimating n-gram probabilities using infini-gram
We estimate the n-gram probability ý of a word w, using the Stupid Backoff scheme (Brants et al. 2007) as described in Equation T where, following Brants al. (2007), set 0.4.
c(w)-n+1)
if ifc(w(/n+)) >
j(ws fw (
o(w)=nt1)" a (00 i-n+1)
otherwise
To calculate unigram probability, in cases where the recursive process described in Equation reaches n 1, also following Brants et al. (007), we calculate j(w)) as described Equation where |C] deseribes the total number of tokens corpus, provided by infini-gram Liu et al. 2024).
f(w:) max c(w,))
|C|
As Brants et al. 2007) note, this approach does provide true probabilities that sum to 1, but these estimates perform similarly well to Dheser-Ney-smoother (Kneser and Ney) 1995) true n-grams, especially on datasets of the scale used in the present work. Additionally, we note that for the unigram counts, we use the word-level counts in the numerator but the token-level counts in the denominator, and thus, unigram probability may be systematically under-estimated relative other n- n-grams. Finally, because our larger n r-grams are calculated at the word level rather than the token level, we expect that our findings may slightly diverge from mechanistic interpretability work looking at token-level relationships (Voita et al. 2024] Chang and Bergen (2025). However, due to the large size of all the datasets involved, unlikely that these factors would drastically impact any of overall patterns identified in this work.
C Contextual Similarity
Given a sequence of words wi...w, the contextual semantic similarity between w, and its context = wi... is defined as the cosine similarity between the w7, the word embedding vector of w, and ć the embedding of the context. A word embedding vector wi is simple the embedding of word as provided using the fastText package (Bojanowski et al., 2017). In our study, we use both the Wikipedia-derie derived fastTText vectors of Bojanowski al. (2017) and the Common- Cwwnon--rraw-deiiiee vectors of Grave et al. (2018). či calculated as in Equation
j=i-1
B W
(3)
With uniform weighting, B I_ With SGPT weighting (Muennighoff 2022), B
19

---

D Evaluation dataset
In order to evaluate how well our heuristics predict language model behavior, we construct a dataset made up Natural Words in Context (NaWoCo), sequences previously unseen by the models. describe the dataset construction process below.
First, we extracted 250,000 sentences from the Fine corpus (Penedo et al. 2024) that had more than 5 words, began with a capitalized word, had no other capitalized words, were assigned a probability of being toxic of 0.1 or less (using the model released by Logacheva et al. (022). and are not in the training data of any of the language models tested -hat is, they were not in the OpenWebTer (Gokaslan and Cohen 2019) or Pile (Gao et al. 2020) corpora determined using infini-gram counts. This last step was to ensure overlap between these sentences and the training data of the models, which could impact model behavior.
From this sample, we then selected 100,000 unique sentences for the training set, and 50,000 each for the validation and test sets. Because we are interested in the probabilities of words in context, we then randomly selected one word (that occurred as the fifth word or later) in each sentence as the critical word, that is, the word for which all metries would be calculated. We then again filtered all these truncated sentences such that they were unique, not toxic, and did not occur in either of method. Finally, we filtered our dataset to
the training corpora using the same previously-descrine ensure that all words were single tokens in the vocabularies of all the language models.
The final WoCo dataset thus made up of a training set of 77,999 items, validation set of 39,474 items, and a test set of 40,980 items.
E Spearman Correlations Between Language Model Log-Probabilities and Predictors
We provide the model-level Spearman correlations between the heuristics and language model log-pnn oog-robability (Figure 3Figure 5)
Open-GPT2 117H
Open-GP12
Parc-Pytha
Parc-Marti 3324
Part WWVV 2239
1.00
0.50
Speammnn Coneldenn Coddiiionn
0.00
2-ggn I togenoo
Pytha TIM
Pytta 201
Fymra 1006
Pyth saoN
3.00
grai Loga vou
<-gren
0:50
grain Logu ols
0.00
Siedartyy (MB) Unevligttd
Fyma
Pytta
Pyte
(MB/SOPT woli todi
1.00
Sements Sindarty
6:50
Samento Siedurts
6.00
122
10
107
22'
10
10
12
10
10
10' 192
10" 22' Training Steps
10
10a
107
10
Figure 3: Spearman correlation coefficient, p between language model log- log-probability and heuristic metrics (n-gram log- probability and word embedding cosine similarity). We show the mean values for all models across seeds and their 95% confidence intervals.
F Seed-Level Correlations Between Language Model Log- and Predictors
We provide the seed-level Pearson (Figure and Spearman (Figure 5) correlations between the heuristics and language model log-probability for all language models.
20

---

Oper GPT2 LETT (oarkraterr
Open GPT2 LLP portletta
Open GPT2 1LPM
Coen 6P112 327M Onnprcca
2343 tburel
wit (bewe))
VETT jeweet
alll roneesineer
Parc Pyllo 203M
Par Pyllo 161M
Porc tyfii IEMM
Forc tytria 160M
Parc Pytra LSOM
Parc Pybli 190M
a8022
Parc Marita 130M
Parc Parka 150M
Pax: Marbs LOON
Pai: Marbs 19OM
Palc Marto 130M
Port Marita 1364
tees)
reed)
seeel
serell
seeds
Pytnia
T/N
1/M
Pytha
Pyths
Pytria LAM
Pyblu LAN
Pydlcc 14M
Pyžio L4M
'ydaa 24M
Heuristc
Pyta
Pyhts
1-gra Laga
Paarson Corrlatton Codtiirk bo
sees)
reed?
seedB
seedi
giai Loga oos
E-grems Epprode
Pytris
TIM
Pythe 31M reed)
Pythi
Pyhis
sewds
seeds
seedl
seeds
grain ots
Samena- fiedactty
(MB/SOPT Welf
Pytri T0M
Pybli TOM
Pyblic N01
Pyllo 7DM
*ydaa 7CM
(CC Uriw plol
Spment: Simdarty TWWeghhee
Pytria seedS
Pybli: ceed?
Pydii seedl
*yhia seeds
seeds
Pythi TEEM seed)
Fytra TEEM reed2
Fyrna 100M seedi
T00M seedi
Pyllio 2.0M
Pytioo 150M
hydia 1000
Fytra 100M
Pytri L00M
Pytto 410M
Pytru
tybra
seedi
Pythi 410M
Fytra V1EM
Fyrns 410M
410M
see0s
reed)
seedl
seeds
Pyttis 10
Fytsa 1.43
Pytta
Pytha C.
Fytta 150
10 12 10
10 38
167
10
12
10
12
10
19
12
10
12
1P
18 10
19
18" 10" UP
16'
10
10
Traiing Stops
Figure 4: Seed-level Pearson correlation coefficient between language model log- rog-probability and heuristic metries (n-gram probability and word embedding cosine similarity).
21

---

Oper GPT2 LEDH coareratert
Open GPT2 LLP portlett
Open GPT2 1LPM
Coen 6PP12 327M Onnprcco
3343 tburel
wit (bewe))
VETT leweet
alll rodeesimeer
Parc Pyllo 203M
Parc Pyfho 161M
Porc tyfii IEEM
Forc tytria 160M
Parc Pytra LOOM
Par Pybli 190M
28022
Parc Mari 130M
Parc Parku 150M
Pax: Marbs LOON
Pai: Marbs 19OM
Palc Marto 130M
Port Marita 1364
sees)
reed)
seeel
serell
seeds
Pytnia
1/N
1/M
Pytha
Pyths
Pytria LAM
Pyblu LAN
Pydlcc 14M
Pyžio L4M
'ydaa 24M
Heuristc
Pyta
Pyhts
1-gra Laga
Speamar Concsaton Conficiott
seesl
reed?
seedB
seedi
giai Loga oos
E-grems Epprode
Pytnis seeds
T1M
Pythe 31M reed)
Pythi
Pyhis
seeds
seedl
seedi
grain ots
Samena- fiedactty
(MB/SOPT Welf
Pytria T0M
Pybli TUM
Pyblic N01
Pydlo 7DM
*ydia 7CM
(CC Uriw plol
Spment: Simdarty TWWeghhee
Pytria seeds
Pybli: ceed?
Pydiu seedl
*yhia seeds
seeds
Pythn TEEM seed)
Fytna TEEM reed2
Fyrna 100M seedi
100M seedi
Pyllio 3.0M
Pytioo 100M
Pydia 1000
Fytra 100M
Pytri L00M
Pytto
Pytra
tytra
seedi
410M
Fytra V1EM
Fyraa 410M seedl
seed5
seeds
reed)
seeds
Pyttis 1t
Fytra 1.43
Pytta
Pytha C.
Fytta 150
12 10
10
107
10
10
12
10
19 12
10
12
10" 1P
18 10l
19
18"
10" UP
16
10
10
10
Traiing Stops
Figure 5: Spearman correlation coefficient p between language model log-probability and heuristic metries (n-gram log- probability and word embedding cosine similarity) the seed level.
22

---

G Cross-Architecture Similarity
We provide the the Pearson correlation r between each seed of each of the models we train (i.e., Parc-Pythia, Parc-Mamba, and Parc-RWKV) at each time step.
213
210
62
011
16
CM
251
16
Figure 6: Pearson correlation r between probabiibilsssss calculated between each model.
23

---

H
Correlations Between Predictors
Figure shows the Pearson correlation coefficient r between all heuristics.
Semantic S imilarity (Wild SGP PPPPT-WWeggge Weightee)
0.34 0.41 0.43
0.42
0.42
0.35 0.42 0.44
0.42 0.41
0.66
0.94
0.7
1
Semantic Similarity 3GPT-Weighed)
0.64
Semantic S iimilariyy (Wik Unweighhed)
0.34 0.36
0.37
0.36
0.36
0.35
0.37
0.38
0.36
0.35
0.67
0.66
0.94
Semantic Similarity (CC Unweighted)
0.69 0.61
0.55
0.5 0.49
0.69
0.61
0.55
0.49
0.46
0.67
0.96
0.66
S-gram Logprobs
0.53
0.7 0.81 0.85
0.85
0.53
0.71
0.85
0.95
0.46
0.35
0.49
0.41
Correlation
The Pile: 4-gram Logprobs
0.58
0.76
0.87
0.89
0.86
0.58
0.77
0.91
0.95
0.49
0.36
0.52
0.42
1.00
3-gram Logprobs
0.86
0.84
0.55
0.44
0.75
The Pile: 2-gram Logprobs
0.8
0.98
0.86
0.78
0.75
0.81
0.87
0.77
0.71
0.61
0.37
0.64
0.42
0.50
The Pile: 2-gram Logprobs
0.99
0.8
0.68
0.61
0.58
0.81
0.68
0.58
0.53
0.69
0.35
0.67
0.35
0.25
@porWebText:: 5-gram Logprobs
0.58
0.76
0.9
0.98
0.75
0.84
0.85
0.49
0.36
0.52
0.42
0.00
0 penWWeblext 4-gram Logprobs
0.61
0.8
0.94
0.98
0.61
0.78
0.88
0.69
0.85
0.5
0.36
0.53
0.42
PerWebtText:: 3-gram Logprobs
0.88
0.9
0.94
0.55
0.43
0 DeenWebTeek Text 2-gram Logprobs
0.81
1
0.88
0.8
0.76
0.8
0.98
0.86
0.76 0.7
0.61
0.36
0.64
0.41
O DenWebTeet: 2-gram Logprobs
0.81
0.68
0.61
0.58
0.99
0.8
0.67
0.58
0.53
0.69
0.34
0.67
0.34
Opersiet 1-grttt--mmmm I-gram --gamtogrii Logantts
OpenWeetToie 2-gram Logrobs
OpenWestort: d-gram Logrots
OpenWesTovs gram Logrobs
Logradas
The Pile: 2ew.:e22-aam.. 2-gram Loganots
The Plle gram Loganbis
The Pile. gram Logaotts
Semarti Sinilely (CC Uniergg saxc)
(WRS Unseightd
Semarde Simiaiity (CC SGPT Woighoo
Semanik Similarty (Wil SCPT ---tlegg (ard)
Figure 7: Pearson correlation coefficient r between all predictor variables.
Seed-level Coefficients
provide the coefficents predicted for each seed of each model for each semantic similarity configuration, namely, SGPT-weighted Wikipedia-based similarity (Figure (8. unweighted Wikipedia- based similarity (Figure D SGPT-weighted Common-Crawll-an similarity (Figure TOp, and unweighted Common-Crawl-ban similarity (Figure

---

Open 6PT2 11TM (berknatee
Oge1- 1P21-GFT2.J1V thaflecttv)
Open GPTZ LLRM
1901-5P12233 1338
(raprical
2343 count)
vis terrel
VITT (eoky1
>El (ceebi mbvort
Parc yyhia 100M
Parc tytli 1OOM
Part Pyllo 103M
Fax tybia 160M
Part Pyllo 153N
Forc Pytlo 16CM
seedi
s0052
Par: Marbu 130M
Port Narto 1312
Fax: Martaa 130M
Parc: Marbo 1392
Fox Mariia 130M
Parc Narbo 130M
seed8
seedl
seed)
seed
seeds
seedt
Pythi
Pytts 14M seedi
L/M
31M
Pytha 1/M
Pyt LAM
Pylia 14M
Pyblu LAN
Pytlii 14M
PyOha LAM
Pyth
seedl
seed?
seedS
sends
Houriste
(Matteed
2-crem Logando noiched
Coaficont
Pyttil
Pytts 31M sends
3TM
31M seedi
Pytha 21M
Loganots (V ariied
teedS
reed?
sends
Sameris Siniarite
Pytii N0N
Pyttia TIM
Pybli 70M
Patlia 7IM
Pydru IVM
Semertic Sinduiity
t-gammmc
Pythi seed5
Pyhhaa seeds
Pydru sends
seed?
seed
TEEM
Pytha 160M
Pytha 1EDM
Pytha 103M
Pysha
Send1236
seedl
seed2
seed
sendi
Pydrio 300M
PAR 100M
Pydti 150M
Pyblu 1AOM
Pydia 200M
Pydiu
Pyblu
M00012204
soud1
sued2
410M
41/2M
Pytha 41DM
Pythis 1106
Pysha 418M seeds
see6S
Sends
see67
soed8
Pythom 10
Pyttis 1:4
Fytra 2.31
Pyrto 9
PyPte '22
10 12
10
10T
38
18
16
10
16
10P 12
107
13
18th 10
18" 16
10" 16 10 38'
10P 10 10? 10
Trainng Stops
Figure 8: Seed-level coefficients for SGPT-weighted Wikipedia- similarity, with 95% confidence intervals
25

---

Open 6PT2 11TM (berknatee
Oge1- 1P21-GFT2.31V thaflecttv)
Open GPTZ LLRM
SP12 1338 (raprical
2343 countt
vit terrel
VITT (eoky1
>El (ceebi mbvort
Parc yyhia 100M
Parc tytli 100M
Part Pyllo 103M
Fax tybia 160M
Part Pydlo 153N
Forc Pytlo 16CM
seedi
s0052
Par: Marbu 130
Port Narto 1312
Fax: Martaa 130M
Parc: Marbo 1392
Fox Mariia 130M
Parc Narbo 130M
seed8
seedl
seed?
seed
seeds
seedt
Pythi
Pytts 14M seedi
L/M
31M
Pytha 1/M
Pyd la LAN
Pylia 14M
Pyblu LAN
Pytlii 14M
PyOha LAM
Pyth
seedl
seed?
seedS
sends
Houriste
(Matteed
2-crem Logando noiched
Coaficont
Pyttil teedS
Pytts 31M sends
3TM
31M seedi
Pytha 21M
Loganots (V ariied
reed?
sends
Sameris Siniarite
Pytii N0N
Pyttia TIM
Pybli 70M
Patlia 7IM
Pydru IVM
Semertic Sinduiity
t-gammmc
Pythi seed5
Pyhhaa seeds
Pydru sends
sees?
seed
TEEM
Pytha 160M
Pythn 1EDM
Pytha 103M
Pysha TEOM
Send1236
seedl
seed2
seed
seedi
Pydrio 200M
PAR 150M
Pydli 150M
Pyblu 1AOM
"ydia 200M
Pydiu
Pyblu
500012204
soud1
sued2
seude
410M
41/M
41DM
Pythi 1106
Pysha 418M seeds
see65
Sends
see67
soed8
Pythom 10
Pytti 1:4
Fytra 2.31
Pyrta 9
PyPte '22
10P 12
10
10
38
18
16
10
16
10P 12
107
13
12th 10
18" 16F
10" 16 10 38
10P 10 107 10
Training Stops
Figure 9: Seed-level coefficients for unweighted Wikipedia-basee similarity, with 95% confidence intervals
26

---

Open 6PT2 11TM (bekknater
Oge1- S1T2-11 thaflecttv)
Open GPT2 LLRM
1901--P12213 1338
(raprical
2343 countt
vis Derrel
VITT (eokyn
>ll (ceebi mbvort
Parc yyhia 180M
Parc tylli 1O0M
Part Pyllo 103M
Fax tybia 160M
Part Pyllo 153N
Forc Pytlo 16CM
seedi
50004
Par: Marts 130
Port Narbo 1312
Fax: Martaa 130M
Parc: Marbo 1392
Fox Mariia 130M
Par: Narbo
reed8
seell
seed)
seed
seeds
seedt
Pythi
Pytts 14M seedi
L/M
31M
Pytha 1/M
s0052
Pyt la LAN
Pyliia 14M
Pyblu LAN
Pytlii 14M
PyOha LAM
seedl
seed?
seedS
sends
Houriste
(Matteed
2-crem Logando noiheed
Couficint
Pyttie seeds
Pytts 33M sends:
3TM
31M seedi
Pytha
Loganoos (V ariied
seed?
sends
Sameris Siniarite
Pythi N0N
Pyttia TIM
Pybli 70M
Patlia 7IM
Pydru IVM
Semertic Sinduuity
t-gammmc
Pythi seed5
PyPhaa seedi
Pydru sends
sees?
seed
TEEM
Pytha 160M
Pythn 1EDM
Pytha 103M
Pysha T88M sendi
Send1136
seedl
see82
seed
Pydiii 200M
PAR 150M
Pydti 150M
Pyblu 1AOM
*ydia 200M
Pydiu
Pyblu
M00d12234
sond1
sued2
410M
41/MM
41DM
Pythi 11/6
Pysha #18M seeds
see6S
Sends
see67
soed8
Pythom 10
Pyttis 3-4D
Fytra 2.31
Pyrto 9
PyPte '32
10 12
10
10T
38
18
16
10
16
10P 32
10?
13
18th 10
18" 16
10" 16 10
10P 10F
107 10
Trainng Stops
Figure 10: Seed-level coefficients for SGPT-weighted Common-Crawl-aee similarity, with 95% confidence intervals
27

---

Open 6PT2 11TM (bekknater
Oge1- 31T2-11 thaflecttv)
Open GPT2 LLRM
10e1-8P12212 1338 (reprical
2343 countt
vis Derrel
VITT (eoky1
>ll (ceebi mbvort
Parc yyhia 180M
Parc tytla 1O0M
Part PyPlo 103M
Fax tybia 160M
Part Pydlo 153N
Forc Pytlo 16CM
seedi
s0052
Par: Mantu 130
Port Narto 1312
Fax: Martaa 130M
Parc: Marbo 1392
Fox Marita 130M
Parc Narbo 1304
reed8
seell
seed)
seed
seeds
seedt
Pythi
Pytts 14M seedi
L/M
31M
Pytha 1/M
s0052
Pyd la LAN
Pylia 14M
Pyblu LAN
Pytlii 14M
PyOha LAM
seedl
seed?
seedS
seeds
Houriste
(Matteed
2-crem Logando noiheed
Couficert
Pyttie teedS
Pytts 31M sends
3TM
31M seedi
Pytha
Loganots (V ariied
reed?
sends
Sameris Siniarite
Pythi N0N
Pyttia TIM
Pybli 70M
Patlia 7IM
Pydru IVM
Semertic Sinduiity
t-gammmc
Pythi seed5
Pyhha seedi
Pydru sends
sees?
seed
TEEM
Pytha 160M
Pythn 1EDM
Pytha 103M
Pysha
Send1236
seedl
see82
seed
sendi
Pydrio 200M
PAR 150M
Pydto 150M
Pyblu 1AOM
"ydia 200M
Pydtu
Pyblu
M00012204
soud1
410M
41/2M
41DM
Pythi 1106
Pytha 418M seeds
see6S
Sends
see67
soed8
Pythom 10
Pyttis 3-4
Fytra 2.31
Pyrto 9
PyPle '32
10 32
10P
10T
10
38
18
16
10
16
10P 12
10?
13
12th 10
18" 16
10" 16 10 38'
10P 10 10? 10
32'
Training Stops
Figure 11: Seed-level coefficients for unweighted Common-Craw/ based similarity, with 95% confi- dence intervals
28

---

Further R analyses
provide plots illustrating the differences in regression R² across seeds for regressions including SGPT-weighted (Figure 12 and mniformly-weighted (Figure 13) similarity.
Ad1 Sopprcca
Open-GFF2
Open GPT2 34OM
-or-----33a 3CM
x543
hovvy
Paic-byth 12M
Fai-ayyth 16tm
Par--yyha 103M
Pan--yyt 163N
PArT-Pyni TEOM
PArt- yol IGEM
Parc-Moonte 1394
Parc Momon LSON
Par Mir--aaii 1S6M
Pax--Mar' LOC
Pac-Mando 1300
Port-Manda 2000
wAde
Pas
COOM
Fan:
ERnCE
redd
seed3
ueeds
see
14M s00F
Pstla
tyPu
PyBle 20V
Pytl 33M
Darssttppl: Toliing
Confeee
Path a 33P
Fyhla 21/
typu 23M
Tydla 23V
PyBl 33M ered)
Regression lufy
read
sim
PyPl seed
Pyla
guam
rendi
see13
sreds
CD and ailllry
stoft
veadi
wxd7
Pybla IGEN
Pynta
Pytts
:GMM
123
Pitl 166M
Pytho I6EM
Pytii HAM
Pytaa SMM
Pytia 193
PAUN ATUM
Tythu AU0
Pybru 4UM
Pa 4:30
89NC2234
seed3
seed4
Pylta
Pytha
tybla
Pybha
caNot
ranK:
seeds
52003254
SeNt1250
seafizas
seeft234
Figure 12: Proportion of the variance in language model log-probability explained by the regressions including weigheed similarity.
29

---

Open
Opnn OFTZ ol coatde
Oyen XOTT (eipana)
20 Donirater
2503
joury')
Pac--yyth
Far--yyta 199M
Par-dyyh tha 143M
Par--yyth 163M
Part-Pyhi
PArC-P
Part-Hammb 133N
Parc -Mtarna 13DM
Parc-Maamaa 1304
Pass Mar--Maaraaa LO0M
Parc-Mramd mrammaammaa 1300
Part- Pxrc-Maama 2306
IHAM
PAC-AMEC 19AM
seed4
PyBau
seedi
Pyth
Fyth
Fybha
Fybla 224
2IM
Dalavel
Paris X11
kyta 11/
kyea 11M
Fyba 11V
Pyel 31M
Regression
Weth lnly
maached by grgaaa
Palia
Pytha 13M
Pybhe 13M
Pydle 1EM
PyBl: IUM
20N02234
seed3
seed4
Pych
Pybha
Pybla eexT
Pyblu te+d
Pytii:
rendi
eneda
ceafl
seell
see15
scelt
16EN
Pytta
Pytts
:00M
123
Pyrta SIEM
Pytta 1LOM
Pyta 110M
kyth 6:30
Pytio--11 41E4
PAtNO HTEM
Pybio 4TEM
Pytio KUM
Pytio 4U/ wad?
Tyteu
"ythu 19nd)334
2yhu2282 rend1336
Tytua 1ead1334
Pyth 12B reed133
K2N63234
32
30
Figure 13: Proportion of the variance in language model log- probabbbbity explained by the regressions including weifhtdlyyweighhedd similarity.
30

---

K Un-normalized Regression Coefficients
While the coefficients of the z-scored variables are easier to compare, they are less straightforward to interpret than those of un-normalized variables. For this reason, we carry out the same analysis with un-normalized variables. To allow the pro-oababiliiies be interpreted bits, we convert them to the form - loga (p). To ensure that all predictors share the same direction, use cosine distance (i.e.,
cosine similarity) rather than cosine similarity.
K.1 SGPT-Weighted contextual semantic distance
We provide the coefficients of the n-gram (Figure 14) and contextual semantic distance (Figure predictors for regressions with SGPT-weighted contextual semantic distance.
5.0
Similaityy Hettic
Sorarii Sinikoitty
Pysk
Fynaa
Pytio
Pyba
Pylio
Sorrandc Sivilurity Meighaed
Cootidort (o (n-gemte aerambbr
Hourish iett:
croo Logarbts
500
oggmmbs
[almakched
Pybhu
Pydtu
Pydiu
Pybru
Pytta
Maathees
Logerabe
00
10
12
10
10'
10 10'
10
10
10
10'
Training Steps
Figure 14: Un-normalized regression coefficients of unigram and 5-gram log- bog-probability over the course of training under different conditions, specifically, whether the n--ram data the same that which the language model was trained (matched) or not (unmatched), and whether SGPT-weighter contextual semantic similarity metric is calculated using Caanon-Crawl-basee or Wikipedia- fastText word vectors.
Cocftciant (oo toctal oosine datance Language rodd 540
Simberty (RUbihdd (Uirechhe pianss
Simieniy
(M (Ma/Way Kort
Pytka
Wegfled!
10
18
307
207
10'
107
101
167
10
12ah Traning Stops
162
207
127
10
10
10
16
107
Figure 15: Un-normalized regression coefficients of contextual semantic similarity over the course of training under different conditions, specifically, whether the n-gram data the same as that on which the language model was trained (matched) or not (unmatched), and whether SGPT-weighted contextual semantic similarity metric is calculated using Common-Crawll-asse or Wikipedia-basee fastText word vectors.
31

---

K.2 Unweighted contextual semantic distance
provide the coefficients of the n-gram (Figure T6p and unweighted contextual semantic distance (Figure 17)
predictors for regressions with unweighted contextual semantic distance.
Porc-Ps
1.2
6-8
Codficcort gambee langagge Food bid
Simlarity Metric
Sorard: Samiluity
Eorari (inelgree))
1.2
Heuristi
Logardee [wlkthee
[annatted
Logands
MMlthee;
E-gean Logeebs
107
107
127
127
121
12'
107
10
10/
107
18"
10f
10
102 10" 10" 10"
107 107
10 10 107
167
101
107
101
107
Training Stops
Figure 16: Un-normalized regression coefficients of unigram and 5-gram log- fog-probability over the course of training under different conditions, specifically, whether the n-gram data the same that on which the language model was trained (matched) or not (unmatched), and whether unweighted contextual semantic similarity metric is calculated using Common Cemmon-Crawl-aee Crawl-based Wikipedia- fastText word vectors.
Open-GP12 112M
Open-GFT2
Par--Pyhas
Parc-Mardaa
Pare-BWee
Codfickn lootootuual cssine detarna hang.ageeeoo mod.edddddd bb)
Simiarly Netric
Pytaa
PyPis
Pytta
Fytta 100m
Pyto, CMM
Samertt Sindarte
Samerts Simburty
Urweightd)
Heuriste
Samarts Simdacty
Pytho
Pytee 14
Pythi
Pytaa 656
Pytlo 128
Simdartt
(Uenothel (-gonns)
10 Traniing
Figure 17: Un-normalized regression coefficients of contextual semantic similarity the course of training under different conditions, specifically, whether the n-gram data is the same as that on which the language model was trained (matched) (unmatched), and whether unweighted contextual semantic similarity metric is calculated using Common-Crawl--ann or Wikipedia- fastText word vectors.
L Language Model Benchmark Performance
evaluate the performance of models used the present study two different ways. First, we calculate perplexity on the seven 'standard language modeling benchmarks' of Paloma (Magnusson
al. 2024), constructed from each the following pre-existing pretraining datasets: C4 (Rffel
et al. 2020), the English subset of mC4 (Chung et al. 2022) Dolma v1.5 (Soldaini et al. 2024).
32

---

Wef (Penedo et al. 2023), Penn Treebank (Marcus et al. 1999). RedPajama (Weber et al. 2024), and WikiText- 103 (Merty al. 2017). We plot the each model's 's perplexity on these text datasets in Figure 18 in bits-per tiss-pe--ytte All evaluations were carried out using the Language Model Evaluation Harness
Gao al.
2021
Biderman et al. 2024).
X322 dortrncee
old Bullooat
ADE Jupecaa
Open SAOM
G712 34x
24T2
Open SIGM
DONEL
BHUCK
Portod wwr-ddarri 19:0P
PACHRENEE USEN
PAI -ESIIO TOMM
PAX Pac-Haorda
-ar---aaaia 133P
LES0
Purs LOEN
DAYY
RMMC
SAM
Pa
DINES
sauc3
Pythia
Taxc
Pepteesy par Bod
(akoma)
Celravas Phlomas
Refradeee shatena
"Hhle 04603
Pyshu
BoNdS
resch
Pers Thekare Palona)
RedPajera Paicea
Ps
Pybli
Pydtia
Pybri
Fysta
rewck
1884
Pyrtin
Pybho
IGEN
Fybha
Pytnis
BINGS
94633
Fyshia
PyRlic
Pyth 1224
Fybla
Pycha
PyBiie
Pych -31M RENCT
Fybla
CMAS
04653
2:80
1ET XP 10" Training Stops
Figure 18: Seed-level perplexities of all models 7 subsets of the Paloma dataset.
We also evaluate their performance of all models five benchmarks. To calibrate benchmark difficulty- if a benchmark is too difficult or too casy, it is not useful for comparing these models-we select
33

---

three benchmarks based on Biderman et al. (2023a) who find that clear differences based on model size and training data can be observed in the performance the Pythia models on the OpenAI version of LAMBADA (Paperno et al. 2016 Radford et al. 2019). SCiQ (Welbl et al.) 2017). and the Easy Set of the A12 Reasoning Challenge (ARC; Clark et al. | 2018). We additionally evaluate the models on SWAG (Zellers et al. 2018) and BLiMP (Warstadt et al. 2020). The accuracy and standard error
of each model on each benchmark is provided in Figure 19
Open 1P 3ITH
Douc 1OPT2 3ITH taafhasarr
Doie CPPT 13:M <777
2043
inber
Parc- yeu LOON
Parc Pyttu LOON
Purts Path COON
Pars the 2004
Pars Pishe 2004 sendi
Fax- -ytt 2004 teeds
BRINC
Parc AWAY 200M
Parc- FAN- 300M
Fax- WIIC SUAM
FUC--MICK SEAM
Par--MMCK 14:M
Part- LEON
Pyltiu LAM
Pyshu LAM
Fysha CAM
Typhi
PVB 24P
Dendhmew
Pythia 331
Fysha 33M
Fyshia 3EM
Pythic 31P
Pyel 31M
pipe sentar
seedi
Pyttis T01
Fyfta Z0M
Fysha T0M
PyPhi TOP
Pyela TOM
PyBns 300M
TARNA LEOR
PatNu BEAM
Pysh LEON
Pydta LOON
KENG
sendi
Pysnos ADOM
Pytnss 433M
Pydia KLEN
SANdIOM
Training
Figure 19: Seed-level accuracies of all models on benchmarks, with shading used to indicate standard error.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly mention contributions in the abstract and introduction, and believe that they reflect our actual contributions in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
35

---

Justification:
No theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide all the code and data necessary reproduce our experiments and analyses the supplementary materials. We also release our trained models with checkpoints.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: provide the data and code necessary to carry out all experiments the supplementary materials, well as instructions for using the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ btiaps..c/public/guue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper itself (Section and Section 4 discusses the necessary details understand the results; and further details are provided in Appendix A Appendix B
Appendix C. Appendix D as well as the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide 95% confidence intervals for main results in Section B For the coefficient analysis in Section 4 we include a breakdown of the results with 95% confidence intervals for each seed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of computational resources used and required in the documentation supplied in our supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
As far
aware, our research conforms to the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We describe acceptable uses of our code, dataset, and trained models in the documentation in our supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We describe acceptable uses of our code, dataset, and trained models in the documentation in our supplementary materials.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the papers (where relevant) and list the licenses of all the assets used in the supplementary materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
39

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide documentation of our code, data, and models, including limitations and licensing information in supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMMs any of the aforementioned ways.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
41


Reviews:
[{"review_id": "YWrZtBBSIE", "reviewer": "Reviewer_8SRQ", "summary": "This work aims to investigate the inner mechanisms of LLMs in feature extraction during training from the perspective of interpretability.  \nBased on the analysis on extensive model checkpoints and datasets, this paper highlights that, no matter the model type or training setup, the behavior of these models at the word level can be largely explained by just three heuristics: word-frequency, n-gram probability, and semantic similarity.\nThey found that these three factors together could explain up to 98% of the variation in the probability that a model assigns to a word in a sentence. Specifically, they observed that as models train, their behavior shifts  in predictable \"phases,\" starting by overfitting to simple patterns (like single-word frequency), then learning more complex n-gram patterns, and becoming sensitive to meaning/semantics early in training.\nMoreover, the work reveals that there exists remarkable and consistent patterns across different architectures (like transformers, RWKV, and Mamba), datasets, and model sizes. \nThis can facilitate our understanding of how LLMs capture statistical features during training and leverage them in prediction.", "questions": "1. This work shows that training patterns are largely consistent across different model architectures. However, in practical applications, LLMs from different families often exhibit varying levels of performance on downstream tasks. How can you explain this discrepancy? Which capability of LLMs can be affected by these patterns?\n2. Figure 3 shows that Pythia models with larger model sizes are less explainable using these three heuristics. Can you further figure out what other significant mechanisms/heuristics (e.g., syntactic features) are at play in larger-sized LLMs? Such mechanisms/heuristics may better explain the performance discrepancies between LLMs across various downstream tasks.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This work investigates the learning behaviors of LLMs from three heuristics, word-frequency, n-gram probability, and semantic similarity, the combination of which can better explain the prediction of LLMs.\n2. As word-frequency and n-gram probabilities based on counts do not take word or phrase similarity in consideration, the introduction of semantic similarity on top of them can better enhance the explainability of LLM prediction behaviors.\n\nWeaknesses:\n1. The analysis focuses on n-grams up to size 5, which may limit the finding of more useful patterns. Language may involve longer-range dependencies, especially in larger, more powerful models.\n2. This work may basically explain how LLMs leverage ngrams for prediction in early training stages. However, as Figure 3 shows, even with these three heuristics, there’s still unexplained variance in model behavior, especially for the largest models and in later training stages, suggesting that more complex or subtle mechanisms are at play.", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "We thank the reviewer for their feedback, and address the weaknesses and questions below:\n\n> The analysis focuses on n-grams up to size 5, which may limit the finding of more useful patterns. Language may involve longer-range dependencies, especially in larger, more powerful models.\n\n> This work may basically explain how LLMs leverage ngrams for prediction in early training stages. However, as Figure 3 shows, even with these three heuristics, there’s still unexplained variance in model behavior, especially for the largest models and in later training stages, suggesting that more complex or subtle mechanisms are at play.\n\n> Figure 3 shows that Pythia models with larger model sizes are less explainable using these three heuristics. Can you further figure out what other significant mechanisms/heuristics (e.g., syntactic features) are at play in larger-sized LLMs? Such mechanisms/heuristics may better explain the performance discrepancies between LLMs across various downstream tasks.\n\nWe acknowledge the reviewer’s concern that there can be dependencies in language longer than those that can be captured by 5-grams, and that our approach leaves some of the variance in model behavior unexplained for larger models later in training. However, our goal in this paper is not to explain all of language model behavior with simple heuristics—if this were possible, there would be no need for contemporary language models in the first place. Instead, our goal in this study is to identify general patterns in the behavior of language models over the course of training across scale and architecture and to see how universal they are. \n\nOur results demonstrate that there is in fact a consistent pattern in the extent to which language model probabilities correlate with the simple heuristics of n-gram probability and word-level semantic similarity, and that after the very early stages of training, the three heuristics we include in our regression (unigram probability, 5-gram probability, and word-level semantic similarity) can explain over 50% of the variance in language model behavior, and at some stages, over 90%. Our hope is that our results can serve as a starting point or a baseline for investigating the more complex patterns of language model behavior that go beyond this.\n\n> This work shows that training patterns are largely consistent across different model architectures. However, in practical applications, LLMs from different families often exhibit varying levels of performance on downstream tasks. How can you explain this discrepancy? Which capability of LLMs can be affected by these patterns?\n\nWe thank the reviewer for this question, and would be interested if the reviewer could provide more details about which specific findings they are referring to. As far as we are aware, in many cases, the differences in performance that can be attributed to architecture alone (rather than, e.g., number of parameters or training data) are not as large as might be expected. For example, while we do see a difference between the performance of Mamba, RWKV, and Pythia models in the original RWKV (Peng et al., 2023) and Mamba (Gu & Dao, 2024) papers, it is not drastic. Additionally, as previously noted, one of our goals in this work is to find commonalities across architectures, which provides a starting point from which to analyze differences.\n\nInspired by this reviewer question, we have carried out further analyses to look at similarities across the models that we trained (i.e., the Mamba, RWKV, and Pythia models trained on OpenWebText) over the course of training. We find that from checkpoint 80 onward, all the log-probabilities of all models are highly correlated—all pairwise comparisons of the Pearson correlation between the log-probabilities calculated by the models have r≥0.93. This again suggests, as in the previous work we note, that in many cases, behavior across architectures is comparable. Returning to the question of differences between architectures observed in previous work, we are not sure which findings precisely the reviewer is referring to, but it is worth noting that studies investigating such questions are likely to explicitly target adversarial and edge cases, and thus, situations in which the differences are larger may simply be rare, and therefore not account for the majority of language model behavior.\n\n**References:**\n- Gu, A., & Dao, T. (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. *First Conference on Language Modeling*. https://openreview.net/forum?id=tEYskw1VY2\n- Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., … Zhu, R.-J. (2023). *RWKV: Reinventing RNNs for the Transformer Era. In H. Bouamor, J. Pino, & K. Bali (Eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023* (pp. 14048–14077). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-emnlp.936"}, {"review_id": "ejktwdGkvp", "reviewer": "Reviewer_xNHd", "summary": "This article analyzes the training dynamics of different architectures in the early stages of training and finds that different architectures will first learn word frequency, n-gram syntax, and context similarity.", "questions": "1. N-grams are a fairly low-level feature, and the intelligence shown by recent large models, such as contextual learning capabilities such as induction heads, is often not in n-grams. I'm not sure how much this helps reveal about the intelligence of the model.  There are some articles about the learning dynamics of n-gram syntax and induction heads for transformers [1][2], which I feel are very relevant to what this paper is about. I think the paper can discuss them.\n\n2. For transformer, mamba and rwkv, their performance in long contexts shows significant differences. However, the indicators measured in this paper seem to be irrelevant to long contexts, so they show greater similarity. What do you think about this?\n\n3. I think the cross-architecture comparison is a highlight of the article, and I also think comparing more powerful architectures would increase the scope of the article. From the perspective of theoretical complexity, transformer, mamba, and rwkv-4 are limited by the expressiveness of TC^0. However, in the past year, some models have broken through the expressiveness limit of TC^0, such as DeltaNet [1], RWKV-7[2], Path[3], and Deltaformer[4], which can perform state tracking tasks such as S5. I am not sure whether such architectures also have learning dynamics similar to those mentioned in this article. \n\n4. I think this article can also discuss broader impacts, such as potential insight for llm training. Starting from this article, do we need to improve the model structure or optimization methods to enable the model to better learn n-grams and quickly move on to the next stage of learning? Or, if an inductive bias for learning n-grams is introduced, it will make the model more inclined to learn n-grams and difficult to move on to the next stage of learning?\n\n\n\n\n[1] Birth of a transformer: A memory viewpoint. NeurIPS.\n\n[2] How Transformers Get Rich: Approximation and Dynamics Analysis. Arxiv 2024.\n\n[3] Linear transformers are secretly fast weight programmers. ICML 2021. \n\n[4] RWKV-7 \"Goose\" with Expressive Dynamic State Evolution. Arxiv 2025. \n\n[5] PaTH Attention: Position Encoding via Accumulating Householder Transformations. Arxiv 2025. \n\n[6] Understanding Transformer from the Perspective of Associative Memory. Arxiv 2025.", "limitations": "See weakness.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The experimental results are detailed. The paper compares the learning dynamics of different architectures and finds many similarities. The cross-architecture comparison fills the gap. \n\n2. The paper is clear and easy to follow.\n\n3. The references are rich, with detailed discussions of related work.\n\n**Weakness**\n\n1. Some details are not clear in the paper.  For example, rwkv and mamba, as core architectures, have had more than one generation of versions, and there are differences between them. The text does not specify which version they are. We have to find out which version is used from the supplementary materials. \n\n2. Novelty is limited. Rather than discovering a novel idea, this paper is more about verifying that an existing idea still holds true in a different architecture.", "was_revised": true, "final_justification": "I think this is a valuable empirical article, but I am not sure if it is sufficient for the NeurIPS. So I tend to weak accept.", "rebuttal": "We would like to thank the reviewer for their detailed questions and feedback, which we address below:\n\n> Some details are not clear in the paper. For example, rwkv and mamba, as core architectures, have had more than one generation of versions, and there are differences between them. The text does not specify which version they are. We have to find out which version is used from the supplementary materials.\n\nWe thank the reviewer for noting this. We will clarify the versions (RWKV-4 and Mamba 1) in the revised paper.\n\n> Novelty is limited. Rather than discovering a novel idea, this paper is more about verifying that an existing idea still holds true in a different architecture.\n\nWe agree that several of the motivating ideas (that language models learn n-grams and predict words that are semantically similar to be more likely than dissimilar words) have been observed and discussed in previous work, but we do believe that our study makes several novel contributions  to our understanding of the training dynamics of language models in addition to showing that the results are consistent across architectures:\n\n1. We show that while models do most closely correlate with n-grams of increasing order over the course of training, the extent to which this is the case decreases with each order.\n\n2. We provide evidence that while language model predictions correlate with n-grams of increasing order over the course of training, their predictions are still biased towards lower-order n-grams relative to these (i.e., unigrams explain language model behavior above and beyond 5-grams). This may partially explain (1), but even if it does not fully explain the pattern, we believe that together these findings provide a valuable starting point for investigating which other heuristics language model predictions may correlate with in the later stages of training, perhaps in combination with mechanistic interpretability analyses of the heuristics we investigate.\n\n3. We provide evidence that except in the very early stages of training, language model predictions correlate with semantic similarity, and the effect of this is relatively consistent throughout training.\n\n4. We provide an estimate of how much variance in language model probability is explained by these simple heuristics, show that it is substantial for all models after the first 10-100 steps (consistently over 50% and up to 98%), and that there is a predictable trend to the amount of variance explained.\n\nAdditionally, we believe that the fact that these patterns are seen despite the differences in architecture is theoretically important in itself, as it may suggest that they reflect more general (or possibly even universal) properties of systems that are trained to predict language incrementally.\n\n> N-grams are a fairly low-level feature, and the intelligence shown by recent large models, such as contextual learning capabilities such as induction heads, is often not in n-grams. I'm not sure how much this helps reveal about the intelligence of the model. There are some articles about the learning dynamics of n-gram syntax and induction heads for transformers [1][2], which I feel are very relevant to what this paper is about. I think the paper can discuss them.\n\nWe thank the reviewer for bringing these papers to our attention. We agree that they are relevant to the current work, and will include a discussion of them (particularly, what is already known about the relationship between global and in-context n-gram prediction and the mechanisms underlying them in transformers) in the revised paper.\n\n> For transformer, mamba and rwkv, their performance in long contexts shows significant differences. However, the indicators measured in this paper seem to be irrelevant to long contexts, so they show greater similarity. What do you think about this?\n\nThis is an interesting point. While there are clear differences between the architectures, it is also true that their behavior is often comparable on downstream tasks (even if there are some differences), as seen in the evaluations reported in the original Mamba and RWKV papers (Gu & Dao, 2024; Peng et al., 2023). Thus, despite their very different architectures, their behavior appears to converge to similar patterns much of the time, which is an interesting implication of our findings (along with previous work) in its own right. With respect to long contexts in particular, the extent to which in practice the capability to deal with very long contexts is important is an open question and likely to vary by domain—for example, it has been argued that there is a general pressure in natural language to have shorter-range dependencies (see, e.g., Futrell et al., 2020); thus, differences in performance on longer-range dependencies may not have as much of an impact on behavior in this domain in practice. In any case, our results help to describe the patterns that occur over shorter contexts (and that appear to be common across models of different architectures), and thus provides a starting point for analyses of longer contexts, other more complex phenomena, and comparisons across architectures.\n\n> I think the cross-architecture comparison is a highlight of the article, and I also think comparing more powerful architectures would increase the scope of the article. From the perspective of theoretical complexity, transformer, mamba, and rwkv-4 are limited by the expressiveness of TC^0. However, in the past year, some models have broken through the expressiveness limit of TC^0, such as DeltaNet [1], RWKV-7[2], Path[3], and Deltaformer[4], which can perform state tracking tasks such as S5. I am not sure whether such architectures also have learning dynamics similar to those mentioned in this article. \n\nWe thank the reviewer for bringing this exciting development to our attention. We agree that a comparison of training dynamics across architectures with different levels of expressiveness would likely be interesting and informative. However, much of the motivation for using such models, as well as the empirical improvements observed, appear to be for long-context settings, and since our analysis focuses on single sentence contexts, we would not expect a substantial difference in the trends observed. Nonetheless, we remain excited in the possibility of such comparisons in future work.\n\n> I think this article can also discuss broader impacts, such as potential insight for llm training. Starting from this article, do we need to improve the model structure or optimization methods to enable the model to better learn n-grams and quickly move on to the next stage of learning? Or, if an inductive bias for learning n-grams is introduced, it will make the model more inclined to learn n-grams and difficult to move on to the next stage of learning?\n\nWe agree that this is an interesting question. If we consider our results for the Pythia models in combination with their downstream performance on tasks as reported in the original Pythia paper (Biderman et al., 2023), it appears that the point at which language model correlation with n-grams of order n≥3 begins to plateau (steps 1,000-2,000; which for the Pythia models corresponds to 2B-4B tokens) is also the point at which they begin to start improving substantially at several downstream tasks: LAMBADA, SciQ, and the Easy set of the AI2 Reasoning Challenge; with the models that end up showing the lowest correlation with n-grams performing the best. Thus, the results of our study could be taken to suggest that learning n-grams may be a prerequisite (but not sufficient) for downstream task performance, and thus, that a training procedure that allows them to be learned more quickly and then move on to the next stage would be the most optimal approach. How exactly this could be implemented is a question for future work. In addition to including a discussion of this in the revision of our paper, we also plan to evaluate all the models included in our study on a wider range of downstream tasks, and thus hopefully gain a better understanding of how the extent to which model performance correlates with the heuristics we look at correlates with downstream performance. In preliminary testing of our models (OWT-Pythia, OWT-Mamba, and OWT-RWKV) on the LAMBADA benchmark, we find that they show the same pattern—their correlation with n-grams of order n≥3 begins to plateau by step 2650, and this is also the point at which we see the improvement at the benchmark begin.\n\n**References:**\n\n- Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., & Wal, O. V. D. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. *Proceedings of the 40th International Conference on Machine Learning*, 2397–2430. https://proceedings.mlr.press/v202/biderman23a.html\n- Futrell, R., Levy, R. P., & Gibson, E. (2020). Dependency locality as an explanatory principle for word order. *Language, 96*(2), 371–412. https://doi.org/10.1353/lan.2020.0024\n- Gu, A., & Dao, T. (2024, August 26). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. First Conference on Language Modeling. https://openreview.net/forum?id=tEYskw1VY2\n- Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., … Zhu, R.-J. (2023). RWKV: Reinventing RNNs for the Transformer Era. In H. Bouamor, J. Pino, & K. Bali (Eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023* (pp. 14048–14077). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-emnlp.936"}, {"review_id": "xVRi65pEWa", "reviewer": "Reviewer_93xh", "summary": "This paper investigates if language model behavior can be explained by simple features like n-gram probabilities and embedding-similarity between tokens and their context. While similar studies have been done in the past, this one expands the scope by studying transformer models of a range of sizes, as well as models of other architectures (Mamba and RWKV). Particularly, the analyses performed in this work are 1) measuring the correlation between LM's log-probabilities and each of the simpler features; and 2) measuring the fit of a regression model that predicts the logprobs from those features, along with the significance of the coefficients. The findings from this work are aligned with prior work.", "questions": "Can you please list some potential actionable insights for LM developers based on the findings in this work?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe scale of the study is larger than prior work, and the results show that the findings generalize across architectures.\n\n### Weaknesses\n\nThe study is very similar to prior work (as is made clear in the paper). While it is nice to see that the findings are aligned with prior work, it is not clear what new actionable insights can be drawn from this work. A discussion on how LM developers can use these findings to improve the LM training procedure would be helpful.", "was_revised": true, "final_justification": "I find the authors' response informative. The discussion has not changed my assessment of the paper.", "rebuttal": "We would like to thank the reviewer for their feedback. We address the weaknesses and questions below:\n\n> The study is very similar to prior work (as is made clear in the paper). While it is nice to see that the findings are aligned with prior work, it is not clear what new actionable insights can be drawn from this work. \n\nOur study highlights that except for in the very earliest stages of training, language model predictions tend to favor tokens with high n-gram probability and word-level semantic similarity. Previous work has demonstrated correlations with these variables, but we show that, at least after the initial 100 or so steps of training, these heuristics can explain 50-98% of the variance in the log-probabilities calculated by language models. This directly relates to previous work that has (implicitly) shown cases where a tendency to make predictions in line with these heuristics leads to undesirable behavior. For example, researchers have found that language models will often complete famous quotes or sayings even when prompted not to (McKenzie et al., 2023), which suggests a strong bias towards frequent n-gram continuations. Similarly, (Gonen et al., 2025) show that language models display ‘semantic leakage’, where they will often generate text that is inappropriately semantically related to words in the input, e.g., “His name is **Cedar**. His friend lives in **a treehouse**” (Llama2-70b-chat completion bolded).\n\nThus, our work provides a more precise and generalizable characterization of the patterns underlying these and related issues with language model behavior, as well as a method for characterizing them in a given model. In terms of actionable insights, then, the analysis methods we demonstrate in the present study may make it possible to predict which models are more susceptible to such biases in their generation behavior before deployment and without relying on more specific tests that are unlikely to cover all possible phenomena. We will include a discussion of this in the revised paper.\n\n\n> A discussion on how LM developers can use these findings to improve the LM training procedure would be helpful.\n\n> Can you please list some potential actionable insights for LM developers based on the findings in this work?\n\nIn combination with the results of research on traditional RNNs, which also become sensitive to the statistics of increasingly long contexts over the course of training (Karpathy et al., 2016), our results suggest that the patterns we observe may reflect more general (and possibly even universal) stages of learning for sufficiently powerful language models of any size or architecture. When we compare our results with work looking at the performance of the Pythia models at downstream tasks at various points over the course of training (Biderman et al., 2023), we see that the models begin to improve at these tasks only after the point at which we see correlations between language model log-probabilities and our heuristics begin to plateau or decrease. This could suggest that learning (at least implicitly) higher-order n-grams and semantic similarity may be a prerequisite for learning the more complex patterns that underlie good performance at downstream tasks; and thus, that speeding up the learning of these heuristics (or even initializing the models such that they align with these heuristics) could lead to more efficient training. This is something we plan to discuss in our revised paper, and plan to further develop by evaluating all the models included in our study on a wider range of benchmarks.\n\n\n**References:**\n- Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., & Wal, O. V. D. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. *Proceedings of the 40th International Conference on Machine Learning*, 2397–2430. https://proceedings.mlr.press/v202/biderman23a.html\n- Gonen, H., Blevins, T., Liu, A., Zettlemoyer, L., & Smith, N. A. (2025). Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models. In L. Chiruzzo, A. Ritter, & L. Wang (Eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)* (pp. 785–798). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.naacl-long.35\n- Karpathy, A., Johnson, J., & Fei-Fei, L. (2016, February 18). Visualizing and Understanding Recurrent Networks. *International Conference on Learning Representations (Workshop Track)*. https://openreview.net/forum?id=71BmK0m6qfAE8VvKUQWB\n- McKenzie, I. R., Lyzhov, A., Pieler, M. M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Shen, X., Cavanagh, J., Gritsevskiy, A. G., Kauffman, D., Kirtland, A. T., Zhou, Z., Zhang, Y., Huang, S., Wurgaft, D., Weiss, M., Ross, A., Recchia, G., … Perez, E. (2023). Inverse Scaling: When Bigger Isn’t Better. *Transactions on Machine Learning Research*. https://openreview.net/forum?id=DwgRm72GQF"}, {"review_id": "GjjcpqFe4X", "reviewer": "Reviewer_Po9Y", "summary": "The authors study the behavior of language models of 3 architectures and different scales as they are trained. They find that frequency of words, n-gram probabilities, and semantic similarly are highly indicative of the predicted probabilities.", "questions": "1) What is the training set used for exactly? Is this only for training the language models or just for the regression? Furthermore, how is it ensured that the sentences in L153-157 are not in any training dataset exactly? \n2) The dataset mentioned focuses on words that are represented as a single token across models. Can we infer that this trend follows to more complex words?\n3) I think a good addition to this work is to include implications of the findings. Would it be worth looking into better initialization on language models based on the insight of these training curves?\n4) L332: On the one hand -> On one hand", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \n1) Well-motivated problem and interesting findings\n2) Good related work and mentions of related work when necessary\n3) Clear description of methodology and approach\n4) Validated findings across a range of model sizes\n\n**Weaknesses**: \n1) I am unsure of the novelty of this work. Is the contribution the systematic training and validation of the correlation between these heuristics, suggested in previous work, and the assigned word probabilities in language models? The value of the introduced dataset is unclear to me.\n2) Please refer to questions below.\n\nI am looking forward to hearing back from the authors and adjusting my scores accordingly.", "was_revised": true, "final_justification": "The authors address my points, and I decide to raise my score to a borderline accept.", "rebuttal": "We thank the reviewer for their comments and questions, which we address below:\n\n> I am unsure of the novelty of this work. Is the contribution the systematic training and validation of the correlation between these heuristics, suggested in previous work, and the assigned word probabilities in language models? \n\nWe believe that the main novel contributions of this study are the following findings:\n\n1. The extent to which models do most closely correlate with n-grams of increasing order over the course of training decreases with each order.\n\n2. While language model predictions correlate with n-grams of increasing order over the course of training, their predictions are still biased towards lower-order n-grams relative to these (i.e., unigrams explain language model behavior above and beyond 5-grams).\n \n3. Except in the very early stages of training, language model predictions correlate with semantic similarity, and the effect of this is relatively consistent throughout training.\n\n4. The variance in language model log-probability explained by three simple heuristics (unigram log-probability, 5-gram log-probability, and word-level semantic similarity) is substantial for all models after the first 10-100 steps (depending on the model), and explains over half (and up to 98%) of their behavior, with the extent to which this is the case over the course of training showing consistent patterns. \n\n5. The above patterns are consistent across size and architecture, suggesting that they may reflect more general or even universal properties of systems that are trained to predict language.\n\n> The value of the introduced dataset is unclear to me.\n\nThe main purpose of our dataset was to provide a set of words in context that are in natural sentences of English and that are not included in the training data of any of the models analyzed.\n\n> What is the training set used for exactly? Is this only for training the language models or just for the regression? Furthermore, how is it ensured that the sentences in L153-157 are not in any training dataset exactly?\n\nThe training set of the introduced dataset was used to fit the regression only, thus allowing us to compare fit on the validation set to ensure that the regression didn’t overfit. We ensured that the sentences in the training set were not included in the model training sets using the infini-gram tool, which allows users to test whether a given text sequence of any length occurs in a dataset. We locally implemented infini-gram for the OpenWebText corpus, and used the built-in API for The Pile corpus. The details of this are provided in Appendix D; however, we agree that this is important and will include them in the main text of the revised paper.\n\n> The dataset mentioned focuses on words that are represented as a single token across models. Can we infer that this trend follows to more complex words?\n\nThis is an interesting question. We constrained our analysis to such words because our language models of interest, n-gram models, and the fasttext models we used all had different tokenizers. We do not have any direct evidence from our own work, but one possibly relevant study is that of Chang et al. (2024), who look at n-gram and transformer language models with the same tokenizers and include tokens that occur as part of larger words. Like us, they find that transformer log-probabilities correlate more with n-gram log-probabilities of increasing order over the course of training, suggesting that at least this pattern is likely to be consistent, but they do find more variability in the log-probabilities assigned to tokens in multi-token words (especially word-intermediate and word-final tokens) across different runs of training. We suspect that the semantic similarity results would similarly be relatively robust but with more variability for tokens in multi-token words.\n\n> I think a good addition to this work is to include implications of the findings. Would it be worth looking into better initialization on language models based on the insight of these training curves?\n\nAs we note in our response to Reviewer 93xh, in combination with previous work the training dynamics of traditional RNNs (Karpathy et al., 2016) and the downstream performance of Pythia models over the course of training (Biderman et al., 2023), our results may suggest that the patterns we observe are prerequisite stages in learning that must occur before the more complex patterns that underlie good performance at downstream tasks can be learned; and thus, that speeding up the learning of n-grams and semantic similarity (or possibly initializing the models such that their predictions reflect these) could in principle lead to more efficient training.\n\n> L332: On the one hand -> On one hand\n\nWe thank the reviewer for noting this typo.\n\n\n**References:**\n\n- Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., & Wal, O. V. D. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. *Proceedings of the 40th International Conference on Machine Learning*, 2397–2430. https://proceedings.mlr.press/v202/biderman23a.html\n- Chang, T. A., Tu, Z., & Bergen, B. K. (2024). Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability. *Transactions of the Association for Computational Linguistics, 12*, 1346–1362. https://doi.org/10.1162/tacl_a_00708\n- Karpathy, A., Johnson, J., & Fei-Fei, L. (2016, February 18). Visualizing and Understanding Recurrent Networks. *International Conference on Learning Representations (Workshop Track)*. https://openreview.net/forum?id=71BmK0m6qfAE8VvKUQWB"}]

Author rebuttal:

Meta-review: This paper investigates interpretability in the training dynamics of LLM across multiple architectures. The central claim is that LLM prediction behavior at the word level can largely be explained by three simple heuristics: word frequency, n-gram probability, and semantic similarity. The authors show that these heuristics together explain up to 98% of the variance in model predictions during training. They also identify predictable training “phases”: models begin with frequency-based prediction, progress to n-gram reliance, and then incorporate semantic similarity. Importantly, these patterns are found to be consistent across architectures and scales, suggesting a potentially universal property of LLM training.

Overall, the paper makes a solid empirical contribution by showing that simple heuristics explain a substantial portion of LLM behavior across architectures. These findings provide a baseline framework for future interpretability studies and suggest possible implications for more efficient training.
Decision: Accept (poster)