Vocabulary embeddings organize linguistic structure early in language model training

## Abstract

Abstract Large language models (LLMs) work by manipulating the geometry of input embedding vectors over multiple layers. Here, we ask: how are the input vocabulary representations of language models structured, and how and when does this structure evolve over training? To answer this question, we use representational similarity analysis, running a suite of experiments that correlate the geometric structure of the input embeddings and output embeddings of two open-source models (Pythia 12B and OLMo 7B) with semantic, syntactic, and frequency-based metrics over the course of training.
Our key findings are as follows: 1) During training, the vocabulary embedding geometry quickly converges to high correlations with a suite of semantic and syntactic features; 2) Embeddings of high-frequency and function words (e.g., “the,” “of”) converge to their final vectors faster than lexical and low-frequency words, which retain some alignment with the bias in their random initializations. These findings help map the dynamic trajectory by which input embeddings organize around linguistic structure, revealing distinct roles for word frequency and function. Our findings motivate a deeper study of how the evolution of vocabulary geometry may facilitate specific capability gains during model training.

## 1 Introduction

Token embeddings are the input vectors to transformer language models. The information that differentiates one input from another, and spurs the diverse and complex processing in large language models, all originates in the vector space of the token embeddings. Understanding the structure of vocabulary embedding representation is therefore a fundamental step in the effort to trace and interpret the internal mechanisms of language models.
In this paper, we analyze the representational space of the token embeddings of 153 Pythia 12-billion checkpoints (Biderman et al., 2023) and 186 OLMo 7-billion checkpoints (Groeneveld et al., 2024), and analyze how the representational relationships in the vocabulary matrix form over the course of training. We track how vocabulary embeddings evolve over training to reflect relationships in 1) semantics, 2) syntax, and 3) word frequency, and how the organizational principles of syntactic class and word frequency affect the convergence of the embedding vectors over training.

Our main analytical approach is Representational Similarity Analysis (RSA; Kriegeskorte et al., 2008; Nili et al., 2014), which measures how correlated the pairwise distance relationships between two representations are. The primary benefit of RSA is that it provides a framework for comparing very different artifacts as long as they are acting on the same stimuli (in our case, the stimuli are English words), and can be defined in terms of a distance metric. RSA lets us compare continuous model vector representations with annotations of the English vocabulary that are in an unrelated medium, such as human word similarity judgments and part-of-speech annotations ([Section 4](#S4)), or word frequency counts ([Section 5](#S5)). Key to our approach is that the English vocabulary is an extremely well-annotated artifact, providing diverse metadata to contextualize the vocabulary space that is the focus of our analyses.

Figure: (a)
Refer to caption: /html/2510.07613/assets/x1.png

We operationalize RSA in two different ways to understand different aspects of the development of vocabulary structure over training. First, we use Hypothesis-Driven RSA (henceforth, ”hypothesis RSA”; Kriegeskorte et al., 2008; Jozwik et al., 2017; Zhang et al., 2017; Groen et al., 2018; Dong et al., 2022; Goddard & Mullen, 2024) to determine when during training and to what degree model representations encode relationships derived from external annotation sources, such as human word similarity judgments or part-of-speech groupings. Second, we examine how significant token groupings (such as tokens of the same part of speech) evolve their intra-group relationships during training to reach their final configuration in the fully trained model, a method we call Convergence RSA. Convergence RSA lets us see how different subsets of the vocabulary change and stabilize during training. This approach is similar in spirit to Representational Trajectory Analysis (Kallmayer et al., 2020), but we study how a given representation evolves throughout training, rather than over the course of model layers.

In Experiment 1, we trace the development of semantic and syntactic structure over training ([Section 4](#S4)). Correlating the vocabulary matrix to datasets of human word similarity judgments (SimLex and WordSim: Hill et al., 2015; Finkelstein et al., 2001; Agirre et al., 2009), as well as syntactic measures (such as part-of-speech or verb class). We find that model vocabulary structure peaks in correlation with these structures at around 15% of training.

In Experiment 2, we trace the effect of word frequency over vocabulary training ([Section 5](#S5)). We show that frequent words stabilize their representations more rapidly, while less frequent words maintain correlations with relationships present in the random initialization. In contrast, the most frequent words completely shed these initialization biases. Moreover, we see that throughout training, the vocabulary matrix progressively develops to represent frequency rank relationships (where words with similar frequency rankings become closer). This reveals how vocabulary representations continue to evolve even after semantic and syntactic measures stabilize early in training.

In Experiment 3, we run a series of analyses to understand what changes after linguistic features stabilize ([Section 6](#S6)). We find that embeddings continue changing rapidly, but that the correlations between input and output embeddings decreases after linguistic stabilization.
We also find a remarkably consistent qualitative result: among all vocabulary words, those experiencing the greatest changes in the last 85% of training are rare (often technical) words moving closer to their morphological inflections.

## 2 Background and related work

### 2.1 Structure in the lexicon

How much information is encoded in a word, and how much in how words are put together? How can we distinguish between lexical representation and grammatical structure, and where does one end and the other begin? These questions are fundamental to understanding how the human and machine language systems operate. Here, we give a brief overview of the complex and meaningful structures that exist in the lexicon.

#### Structure in the vocabulary

While many syntactic analyses describe abstract grammatical systems like recursion that operate independently from a speaker’s lexical semantic knowledge (see Hauser et al. (2002); Pinker (1998); Chomsky (1994) for some notable examples of this “words and rules” approach), several theoretical frameworks highlight the rich syntactic information embedded within the lexicon and its interface with grammar. One influential alternative perspective is frame semantics: the idea that words contain information about which kinds of words can go with them, or what frames they can appear in
(Fillmore et al., 2006; Baker et al., 1998; Levin, 1993; Kipper et al., 2008).
Pustejovsky (1998) proposes that the lexicon is highly generative, accounting for subtle related meaning variations (such as “fast” meaning going quickly, or, lasting a brief time)
and creative extensions of word meaning (as in “Corvetted across the USA”) (Clark & Clark, 1979).
Lastly, diverse work explores the idea of the syntax-lexicon continuum (Croft, 2001; 2020; Tomasello, 2005; Jackendoff, 2011; Goldberg, 1995): that there is no clear separation between single words, semi-fossilized structures like “not only…but also”, and more general grammatical patterns like the passive voice. Following these analytical ideas, we examine linguistic structure in the vocabulary, contributing to a program of interpretability that does not a priori limit what information is in what part of the model.

#### Frequency in the vocabulary

One fundamental fact across human languages is that their vocabularies follow a Zipfian (or, power-law) distribution (Zipf, 1936; Mandelbrot, 1953), where there are few words that make up the bulk of words in any given sentence (“the”, “of”), and many words with very low frequency (“hedgehog”, “popsicle’) (for a comprehensive review of Zipfian vocabulary distributions and related theories, see Piantadosi (2014)).
The power-law distribution of the vocabulary is one of its basic structuring principles, and here we investigate how the vocabulary matrix interfaces with it in [Section 5](#S5).

### 2.2 The embedding matrix in LLMs

Most transformer language models encode word meaning in a word embedding matrix of dimension $(V\times d)$, where $V$ is the size of the tokenizer vocabulary, which splits most text into units that roughly represent meaningful words (in many language models $V$ is around 50-100 thousand), and $d$ is the model dimension. Each token in the tokenizer is assigned a vector in the embedding matrix that represents it,
and a row of the embedding matrix cannot receive any gradient from backpropagation unless the specific token has appeared in the current batch (some exceptions to this dominant tokenization paradigm include Clark et al. (2022), Xue et al. (2022), Ahia et al. (2024), Kallini et al. (2025), Huang et al. (2023), and Feher et al. (2024)). In most standard transformer models, all subsequent activations are derived by manipulating and establishing relationships between the initial vectors from the embedding matrix.

While substantial interpretability research has focused on model activations in later model layers,
and early foundational interpretability work focused on understanding the geometric structure of static word embeddings like word2vec
(see Mikolov et al. (2013); Pennington et al. (2014); Ethayarajh et al. (2019); Hashimoto et al. (2016); Allen & Hospedales (2019)), and a line of work looks at more general model training dynamics Chen et al. (2023a); Tirumala et al. (2022),
research examining the embedding matrix of language models from the training perspective remains comparatively sparse.
Past work has shown complex lexical representations even in models with insufficiently expressive tokenizers (Feucht et al., 2024), tracked the evolution of grammatical interpretations from word embeddings to later layers (Papadimitriou et al., 2022), analyzed how LM gradients project onto the vocabulary matrix (Katz et al., 2024), and examined how rare undertrained tokens affect model behavior (Land & Bartolo, 2024). Others have also explored how vocabulary re-learning influences cross-lingual transfer (Wu et al., 2023; Patil et al., 2022; Chronopoulou et al., 2020; Chen et al., 2023b),
the effects of using vastly different sizes of vocabulary matrices (Liang et al., 2023; Schmidt et al., 2024),
and the role of vocabulary in scaling language models (Huang et al., 2025; Wies et al., 2021). Our work introduces a comprehensive framework for representational analysis that lets us systematically analyze the encoded linguistic structure in embeddings across training.

### 2.3 Hypothesis-driven Representational Similarity Analysis

Representational Similarity Analysis (RSA) quantifies how stimuli are encoded by comparing their evoked response patterns (Kriegeskorte et al., 2008; Nili et al., 2014). By constructing Representational Dissimilarity Matrices (RDMs) from pairwise comparisons of multivariate responses (from fMRI, neural recordings, or model activations (Lepori & McCoy, 2020)), RSA enables correlation between different representational systems containing different numbers of features. In hypothesis-driven RSA, theoretical accounts are translated into interpretable RDMs for direct comparison with empirical data, allowing researchers to test whether specific regions or networks organize information by semantic categories or other metrics (Jozwik et al., 2017; Groen et al., 2018). For instance, animacy RDMs can help determine if visual cortex encodes animate/inanimate distinctions (Nili et al., 2014). We apply this approach to track how vocabulary representations in language models evolve throughout training, examining both their alignment with linguistic features and the developmental trajectories of semantically and syntactically defined word groups.

## 3 Methods

We next outline the key methodological choices underlying all experiments, with dataset details and experiment-specific implementations described in methods sections corresponding to each analysis.

#### Models and embedding matrices

We use the 153 released training checkpoints of the 12-billion-parameter Pythia-12B model (Biderman et al., 2023), and 186 released checkpoints of the OLMo-7B model (Groeneveld et al., 2024) (this is every third checkpoint of the 558 released checkpoints, due to resource constraints). In all, we run the analyses outlined in this paper on a total of 339 separate language model instances. We run experiments on the input and output embeddings of the models, which are not tied in both Pythia and OLMo (i.e., they are separate embeddings of the lexicons). Though understanding the effect of tying the input and output embeddings would be interesting (Press & Wolf, 2017; Bertolotti & Cazzola, 2024), we are not aware of any large, modern models that open-source their training checkpoints and have weight-tying. For brevity, we only show Pythia input embeddings results in the figures in the main text. Results for OLMo can be found in [Appendix A](#A1), and results for output embeddings in [Appendix B](#A2). Both sets of findings are discussed in the main text where relevant.

The Pythia and OLMo models share an identical tokenizer vocabulary (and embedding matrix length) of 50,688, of which 28,000 are full words and word starts, with the rest being subword continuation tokens. Throughout our analyses, we focus on full word tokens: tokens that are in exact correspondence with the words of our annotated datasets. Understanding the dynamics and geometry of subword tokens remains an interesting avenue for future work.

#### Distance in model embedding space

Throughout the analyses in this paper, we use Spearman distance between embedding vectors as a distance measure, which indicates whether the dimensions of the two vectors have the same ranking, regardless of magnitude. This follows the best pratices suggested by Zhelezniak et al. (2019), and the results of Timkey & van Schijndel (2021), who found that oversized dimensions with little causal influence on model behavior can distort other representational distance measures. To ensure our choice of measure does not substantially affect results, we replicated the semantic experiments from Experiment 1 using both Euclidean and cosine distance, obtaining nearly identical outcomes (see [Appendix C](#A3), [Figure 9](#A3.F9)).

#### Hypothesis RSA: correlation with annotated features ( Figure 1(a) )

To run our hypothesis-driven analyses, we take the vocabulary matrix of dimension $V\times d$ and create a pairwise distance matrix (a representational dissimilarity matrix, RDM) $D_{\text{model}}\in\mathbb{R}^{V\times V}$, using the Spearman distance between the embeddings corresponding to each pair of words. Then, for each annotated hypothesis dataset, we construct a dissimilarity matrix over the subset of words appearing in the dataset, yielding $D_{\text{hypothesis}}\in\mathbb{R}^{V^{\prime}\times V^{\prime}}$, where $V^{\prime}<V$. To compare with the model, we extract the corresponding $V^{\prime}$ rows and columns from $D_{\text{model}}$, producing $D^{\prime}{\text{model}}\in\mathbb{R}^{V^{\prime}\times V^{\prime}}$. We then compute Kendall’s $\tau$ correlation between the vectorized lower triangular entries in $D_{\text{hypothesis}}$ and $D^{\prime}_{\text{model}}$, which quantifies the representational similarity between the vocabulary matrix and the annotated hypothesis. This procedure is repeated for every model checkpoint to track how similarity evolves over training.

#### Convergence RSA: correlation with final training checkpoint ( Figure 1(b) )

In Convergence RSA, instead of comparing the structure of the vocabulary matrix against external hypotheses, we measure how the representational geometry at each checkpoint correlates with that of the final trained model. For a given subset of tokens $V^{\prime}$, we create a representational dissimilarity matrix $D_{\text{model},i}^{\prime}\in\mathbb{R}^{V^{\prime}\times V^{\prime}}$ at training step $i$ using pairwise Spearman distances. We then compute Kendall’s $\tau$ correlation between $D_{\text{model},i}^{\prime}$ and $D_{\text{model},\text{final}}^{\prime}$, yielding a measure of similarity between the representation at step $i$ and the fully-trained representation. This approach allows us to track how different subsets of the vocabulary converge to their final representational structure over the course of training.

## 4 Experiment 1: Hypothesis RSA with semantic and syntactic measures

Figure: (a) Method example matrices
Refer to caption: /html/2510.07613/assets/x3.png

Our first investigation focuses on understanding when model token embeddings represent relationships of meaning and language structure, using annotated semantic and syntactic measures .

### 4.1 Methods: Semantic and syntactic measures

#### Semantic Similarity Measures

For the semantic similarity measures, we use word-pair similarity annotations.
Since semantic similarity measures contain sparse human annotations between words (i.e, not every possible word pair has an annotation associated with it), we cannot correlate with the full vocabulary RDM, and instead report correlations with the vector corresponding to the embedding distances between the annotated pairs. The datasets we use are:

1) the WordSim-353 Dataset (Finkelstein et al., 2001) contains 353 word pairs with similarity ratings averaged across 13-16 human subjects.
2) The SimLex-999 dataset (Hill et al., 2015) contains 999 word pairs, covering nouns, adjectives, and verbs, annotated by 500 subjects. For both WordSim and SimLex, our reported correlations include only those word pairs where both words exist in the Pythia/OLMo tokenizer.
3) GloVe word embedding vectors
GloVe vectors (Pennington et al., 2014) represent word meaning in a high-dimensional space, encoding important semantic relationships like analogies
linearly. We take the set of 983 word pairs from SimLex that’s common across the GloVe vocabulary and the Pythia/OLMo tokenizer, so the GloVe results are comparable with the word similarity datasets.

#### Syntactic Similarity Measures

For the syntactic similarity measures, we take syntactic lexicon annotations and make hypothesis RDMs out of them. For example, for part-of-speech annotations, the hypothesis distance matrix is that words with the same part of speech have distance 0 and words with different parts of speech have distance 1. We emphasize that each hypothesis is relatively simple, so we do not expect high absolute correlation values, as those would imply that embeddings encode only a single basic relation like part of speech. Instead, the correlations are best interpreted in a comparative sense, either across hypotheses or across training stages, rather than as absolute measures of encoding strength. We detail our syntactic hypothesis RDMs below:

1) Part of speech RDM The first syntactic feature that we test is part of speech: the syntactic classes like verbs and nouns. We use the human-corrected EWT Universal Dependncies treebank parses (Silveira et al., 2014; de Marneffe et al., 2021; Nivre et al., 2020), and if a word appears as a part of speech at least 5 times it receives distance 0 to all other words with that part of speech. 2) Wiktionary Tags RDM For a fine-grained set of features, extract the Wiktionary tags of words (syntactic tags such as transitive or regional/register tags such as informal), and create an RDM where words that share a tag have distance 0 (we use Ylonen, 2022, Wiktextract). We subsampled 3,000 words (of the 21,500 possible words) for constructing our RDM in order to make pairwise distance calculations computationally feasible. We ran thhe experiment 5 times and saw only small variations (standard deviation of $<$ 0.23 throughout every training checkpoint).
3) VerbNet classes RDM We create a hypothesis RDM where verbs of the same VerbNet verb class have distance 0. VerbNet (Korhonen & Briscoe, 2004; Kipper et al., 2006; 2008) is a comprehensive resource carefully annotated by experts that categorizes verbs into verb classes (Levin, 1993): sets of verbs that take the same verb arguments and the same alternations.
4) Random Baseline As a conservative control, we assess how model embeddings correlate with a ‘grouping’ metric that lacks any syntactic information. This baseline consists of a hypothesis RDM with the same number of classes as VerbNet and the same number of words per class, but with randomly sampled tokens in each class. The correlation with this RDM gives a baseline for how much token embeddings correlate with distances that correspond to consistent groupings of words, regardless of whether these groupings are in some way meaningful.
5) Part of Speech and Wiktionary RDM
Lastly, we evaluate model correlations to a dissimilarity matrix that combines information from part of speech and Wiktionary. In this hypothesis RDM, distances reflect graded overlap: two words receive a distance of 0 if they share both tags, 0.25 if they share only part of speech, and 0.5 if they share only a Wiktionary tag. This scheme heuristically encodes the intuition that joint agreement is stronger than either source alone.

### 4.2 Results

#### Semantic information arises early in training

We present our results in [Figure 2](#S4.F2). Overall, [Figure 2(b)](#S4.F2.sf2) shows that the token embeddings of the model converge to their approximate final correlation relatively quickly — within 10,000 steps for correlation with SimLex distances. This indicates that the semantic structure of vocabulary representations is established very early in training. Results for OLMo and for output embeddings are very similar.
Results for OLMo are very similar, suggesting that this early emergence of semantic structure is not specific to a particular architecture or training setup but may be a more general property of LMs. We also find that output embeddings behave similarly, indicating that the semantic organization is shared across both input and output vocabularies. All together, our findings imply that the rapid emergence of semantic structure is both model-agnostic and robust across different embedding roles.

Interestingly, we also observe that model embeddings converge to encode similarity much more than relatedness (see [Appendix D](#A4), [Figure 10](#A4.F10)). The Similarity and Relatedness splits of WordSim (Agirre et al., 2009) disambiguate similarity (words like ‘car’ and ‘truck’) and relatedness (words like ‘car’ and ‘road’, which are related but very dissimilar objects). We find that at the final checkpoint, model embeddings are significantly more correlated with the WordSim-Similarity split (Pythia correlation 0.56) than with the WordSim-Relatedness split (Pythia correlation 0.21), a distinction which static embeddings fail to make (the anaologous GloVe difference is 0.59/0.50).

#### Syntactic organization of the vocabulary peaks early in training

The syntactic results in [Figure 2(c)](#S4.F2.sf3) show that model embeddings converge to their final correlations with our syntactic organization hypotheses early in training. In fact, for all three of our syntactic RDMs (part of speech, Wiktionary tags, and VerbNet classes), the correlation peaks early and then stabilizes to a lower value than the peak. We see similar results in OLMo, with one key difference: though the POS embeddings have a sharp elbow at a similar point in training, and at a similar correlation value to Pythia, they then continue rising. This suggests an interesting direction for future work: understanding why OLMo attains stronger POS correlations than Pythia, even when other results are nearly identical.
One notable exception to the Pythia early-peak pattern is also the best-correlating hypothesis: the combination of part-of-speech and Wiktionary tags. Correlation to this joint hypothesis RDM grows more monotonically and stabilizes slightly later than any of the other hypotheses. This points to the possibility that the relationships that arise in model embeddings over training are better explained by nontrivial combinations of linguistic features, and understanding these interactions is an interesting avenue for future investigation in interpretability.
In sum, we find that semantic structure emerges rapidly and achieves the strongest correlations overall, while syntactic organization peaks early at lower levels, with the best performance coming from combined linguistic feature hypotheses.

## 5 Experiment 2: The effect of frequency

Figure: (a) Convergence RSA by frequency buckets OLMo in [Figure 6](#A1.F6), Unembeddings in [Figure 8](#A2.F8)
Refer to caption: /html/2510.07613/assets/x6.png

A fundamental and far-reaching aspect of language is that the vocabulary of human languages follows a power-law distribution.
Here, we examine how the vocabulary matrix encodes the structural effects of vocabulary frequency over training.

### 5.1 Methods

Convergence RSA by frequency buckets
As in our part-of-speech analysis, we test how quickly different splits of the vocabulary converge to the final representations at the end of training. In order to test the effect of frequency, we split the top 1000 words in the vocabulary into 10 frequency buckets, and run Convergence RSA for each bucket, as illustrated in [Figure 1(b)](#S1.F1.sf2).

Hypothesis RSA: correlation to frequency rank and frequency count
We also use frequency measures to construct two hypothesis RDMs: the Frequency-Rank and the Frequency-Count dissimilarity matrices. In the Frequency-Rank RDM, the distance between two words is how far apart they are in the frequency rank: the most frequent word and the 3rd-most-frequent word have a distance of 2. In the Frequency-Count RDM, the distance between two words is the difference in their frequency counts, and so they would have a distance of 2.3 billion, owing to the extreme disparity in occurrence counts between the most common and moderately common words in the distribution.

To make our frequency counts applicable across multiple models (as it is very expensive to count the frequency of words over a large corpus), we use a model-agnostic enhanced space tokenization regex and get frequency counts from C4, described in more detail in [Appendix E](#A5).

### 5.2 Results

#### High frequency words converge quickly

As shown in [Figure 3(a)](#S5.F3.sf1), high-frequency words (blue) stabilize faster in training than low-frequency words (orange), with the top 100 most frequent words showing significantly faster convergence than even the next bucket of 100 words. However, rescaling the x-axis by expected exposures per bucket reveals a different picture: high-frequency words change more slowly with each exposure, while low-frequency words shift more per occurrence and converge quickly despite fewer total exposures. Our results expose a dual effect of frequency: while high-frequency words stabilize earlier in absolute training time due to their frequent occurrence, each individual exposure produces smaller changes to their embedding vectors compared to low-frequency words. We show a related effect in [Figure 11](#A6.F11), [Appendix F](#A6), where function words (grammatical like “the”, which are also generally higher-frequency) converge faster than lexical words (words with specific meanings like “cat”). Due to the high correlations of these features, we cannot know if an effect is due to frequency or function.

#### Low-frequency words are correlated with their random initializations

In the inset of [Figure 3(a)](#S5.F3.sf1), we see that most lower-frequency words converge to representational structures that maintain correlation with their random initialization patterns. Less frequent words preserve stronger connections to their initialization vectors even at training completion. Only the most common 100 words fully break from their random initialization, showing 0 correlation between final embeddings and the starting vectors. This suggests that high-frequency tokens receive sufficient gradient updates to completely reshape their embedding structure, while rare words are biased by artifacts of their random initialization. OLMo results show the same patterns.

#### Tokens move towards their frequency rank gradually throughout training

As shown in [Figure 3(b)](#S5.F3.sf2), when we construct a hypothesis RDM based on frequency rank, we observe a gradual, monotonic increase in correlation across training. This indicates that the embedding matrix continues to evolve, with tokens progressively clustering according to their relative frequency positions (e.g., frequent vs. rare words). In contrast, when we use raw frequency counts, convergence is highly non-monotonic, implying that absolute occurrence counts do not provide a stable organizing principle. Taken together, these results suggest that the model embeds vocabulary in a way that reflects relative frequency rank rather than raw frequency, producing a more robust representational structure that strengthens steadily over training.

## 6 Experiment 3: How do embeddings change after linguistic features stabilize?

As Experiment 1 shows, many key linguistic features reach their peak correlations very early in training. This raises a central question: after roughly 15% of training, do embeddings continue to change, or do they remain largely static? If they were static, the absence of further change in linguistic features would be expected. We next present three analyses demonstrating that embeddings do, in fact, continue to evolve well beyond the point at which linguistic features appear to stabilize.

Figure: (a)
Refer to caption: /html/2510.07613/assets/x8.png

#### Embeddings keep changing after linguistic features stabilize

In [Figure 4(a)](#S6.F4.sf1), we plot the average raw distance between 1,000 randomly sampled token embeddings at each checkpoint and their corresponding embeddings at the final training step. Unlike our RSA analyses, this measure does not capture relational geometry between tokens, but simply how far individual embeddings have moved in absolute terms. The results show that embeddings continue to shift substantially even after 20,000 steps, well beyond the point where correlations with linguistic features appear to have stabilized.

#### Correlation between input and output embeddings peaks with linguistic representations

In [Figure 4(b)](#S6.F4.sf2), we plot the correlation between input and output embeddings. Although Experiments 1 and 2 showed that both sets of embeddings correlate similarly with linguistic features, this does not imply that they are correlated with each other. Indeed, we find relatively low RSA correlation between their representational spaces overall. Notably, however, a clear peak appears at roughly 15% of training—the same point at which correlations with linguistic features peak. When bucketing by frequency, we also find a strong effect: high-frequency words (light blue) show substantially stronger correlations between input and output embeddings than low-frequency words.

#### Qualitative analysis: The words that get closest after stabilization are inflections of rare nouns

For a qualitative investigation of which tokens change the most after the first 20,000 steps, we calculate two $V\times V$ dissimilarity matrices: one at 20,000 steps and one at the final checkpoint (142,000 steps). We then take their difference, $D_{\text{diff}}=D_{\text{final}}-D_{\text{20K}}$, and examine the maximum and minimum values — that is, the word pairs that experience the greatest changes. We find that the largest decreases in distance (embeddings moving closer) are substantially greater in magnitude than the largest increases (embeddings moving farther), with a maximum difference of 0.476 vs. 0.176 in Spearman distance. The word pairs that have the greatest increase in representational similarity are a strikingly consistent set: rare, technical nouns and their plurals [Figure 4(c)](#S6.F4.sf3). We hypothesize that bringing words close to their inflected forms is a proxy effect of learning their meaning, since these forms have nearly identical semantics. By contrast, the pairs that have the greatest decrease in similarity are common words like “of” with word fragments like “teasp”, as shown in [Figure 12](#A7.F12), [Figure 13](#A7.F13).
Both of our qualitative findings corroborate our frequency results from [Section 5](#S5), highlighting frequency as a primary factor in driving embedding changes.

## 7 Discussion and limitations

We have used representational similarity analysis to examine how vocabulary embeddings in language models evolve during training, revealing that semantic and syntactic structures emerge early while frequency effects have continuous influence on the geometry. Later training primarily refines morphological relationships between rare words, showing how vocabulary representations organize around linguistic structure with distinct roles for word frequency and function. Since frequency has such far-reaching effects in the lexicon, a key limitation is that our current experimental paradigm cannot distinguish between the effects of frequency and some of the other factors that we test. Therefore, a promising avenue for future work would be to try and isolate signatures in lexicon training dynamics that are more provably effects beyond the role of frequency. One way to do this would be to track the representational changes of each word individually (as opposed to grouped into meaningful clusters), and then tease out the effects of frequency versus other features.

Analyzing and understanding the embedding matrix of LLMs is also an important step in linking LLMs to human language cognition. Firstly, embedding matrices raise an interesting question: does the methodological choice of a large token embedding matrix in LLMs implicitly build in the assumption of a ‘words-and-rules’ (Pinker, 1998) approach to language into the system? Most LLMs indeed separate these two mechanisms conceptually, between the initial embeddings that represent words, and the subsequent model components that manipulate them. At the same time, studying the training dynamics of embeddings provides a window into how different aspects of language learning may be bootstrapped. We find clear evidence that nontrivial grammatical and semantic structure is emergent within the initial high-dimensional embedding spaces of LLMs, shaping how the lexicon interacts with later layers. The aim of this paper has been to take first steps toward characterizing these lexicon representations: how they encode structural properties of language, and how such organization emerges over training.

## References

- Agirre et al. (2009)
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa.
A study on similarity and relatedness using distributional and WordNet-based approaches.
In Mari Ostendorf, Michael Collins, Shri Narayanan, Douglas W. Oard, and Lucy Vanderwende (eds.), *Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pp. 19–27, Boulder, Colorado, jun 2009. Association for Computational Linguistics.
URL [https://aclanthology.org/N09-1003/](https://aclanthology.org/N09-1003/).
- Ahia et al. (2024)
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov, and Noah A Smith.
Magnet: Improving the multilingual fairness of language models with adaptive gradient-based tokenization.
*Advances in Neural Information Processing Systems*, 37:47790–47814, 2024.
- Allen & Hospedales (2019)
Carl Allen and Timothy Hospedales.
Analogies explained: Towards understanding word embeddings.
In *International Conference on Machine Learning*, pp. 223–231. PMLR, 2019.
- Baker et al. (1998)
Collin F Baker, Charles J Fillmore, and John B Lowe.
The berkeley framenet project.
In *COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics*, 1998.
- Bertolotti & Cazzola (2024)
Francesco Bertolotti and Walter Cazzola.
By tying embeddings you are assuming the distributional hypothesis.
In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pp. 3584–3610. PMLR, 21–27 Jul 2024.
URL [https://proceedings.mlr.press/v235/bertolotti24a.html](https://proceedings.mlr.press/v235/bertolotti24a.html).
- Biderman et al. (2023)
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.
Pythia: A suite for analyzing large language models across training and scaling.
In *International Conference on Machine Learning*, pp. 2397–2430. PMLR, 2023.
- Chen et al. (2023a)
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra.
Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms.
In *The Twelfth International Conference on Learning Representations*, 2023a.
- Chen et al. (2023b)
Y Chen, K Marchisio, R Raileanu, DI Adelani, P Stenetorp, S Riedel, and M Artetxe.
Improving language plasticity via pretraining with active forgetting.
In *Advances in Neural Information Processing Systems*. NeurIPS, 2023b.
- Chomsky (1994)
Noam Chomsky.
*The Minimalist Program*.
MIT Press, 1994.
- Chronopoulou et al. (2020)
Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser.
Reusing a pretrained language model on languages with limited corpora for unsupervised nmt.
In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 2703–2711, 2020.
- Clark & Clark (1979)
Eve V. Clark and Herbert H. Clark.
When nouns surface as verbs.
*Language*, 55(4):767–811, 1979.
ISSN 00978507, 15350665.
URL [http://www.jstor.org/stable/412745](http://www.jstor.org/stable/412745).
- Clark et al. (2022)
Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting.
Canine: Pre-training an efficient tokenization-free encoder for language representation.
*Transactions of the Association for Computational Linguistics*, 10:73–91, 2022.
- Croft (2001)
William Croft.
*Radical construction grammar: Syntactic theory in typological perspective*.
OUP Oxford, 2001.
- Croft (2020)
William Croft.
*Ten lectures on construction grammar and typology*, volume 11.
Brill, 2020.
- de Marneffe et al. (2021)
Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman.
Universal dependencies.
*Computational Linguistics*, 47(2):255–308, 07 2021.
ISSN 0891-2017.
doi: 10.1162/coli˙a˙00402.
URL [https://doi.org/10.1162/coli_a_00402](https://doi.org/10.1162/coli_a_00402).
- Dong et al. (2022)
Haibin Dong, Na Li, Lingzhong Fan, Jianguo Wei, and Junhai Xu.
Integrative interaction of emotional speech in audio-visual modality.
*Frontiers in Neuroscience*, 16:797277, 2022.
- Ethayarajh et al. (2019)
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.
Towards understanding linear word analogies.
In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp. 3253–3262, 2019.
- Feher et al. (2024)
Darius Feher, Benjamin Minixhofer, and Ivan Vulić.
Retrofitting (large) language models with dynamic tokenization.
*arXiv preprint arXiv:2411.18553*, 2024.
- Feucht et al. (2024)
Sheridan Feucht, David Atkinson, Byron C Wallace, and David Bau.
Token erasure as a footprint of implicit vocabulary items in llms.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 9727–9739, 2024.
- Fillmore et al. (2006)
Charles J Fillmore et al.
Frame semantics.
*Cognitive linguistics: Basic readings*, 34:373–400, 2006.
- Finkelstein et al. (2001)
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin.
Placing search in context: The concept revisited.
In *Proceedings of the 10th international conference on World Wide Web*, pp. 406–414, 2001.
- Goddard & Mullen (2024)
Erin Goddard and Kathy T Mullen.
Temporal evolution of color representations measured with meg reveals a ‘coarse to fine’dynamic.
*bioRxiv*, pp. 2024–12, 2024.
- Goldberg (1995)
Adele E Goldberg.
*Constructions: A construction grammar approach to argument structure*.
University of Chicago Press, 1995.
- Groen et al. (2018)
Iris IA Groen, Michelle R Greene, Christopher Baldassano, Li Fei-Fei, Diane M Beck, and Chris I Baker.
Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior.
*Elife*, 7:e32962, 2018.
- Groeneveld et al. (2024)
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.
Olmo: Accelerating the science of language models.
*Preprint*, 2024.
- Hashimoto et al. (2016)
Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola.
Word embeddings as metric recovery in semantic spaces.
*Transactions of the Association for Computational Linguistics*, 4:273–286, 2016.
- Hauser et al. (2002)
Marc D Hauser, Noam Chomsky, and W Tecumseh Fitch.
The faculty of language: what is it, who has it, and how did it evolve?
*science*, 298(5598):1569–1579, 2002.
- Hill et al. (2015)
Felix Hill, Roi Reichart, and Anna Korhonen.
Simlex-999: Evaluating semantic models with (genuine) similarity estimation.
*Computational Linguistics*, 41(4):665–695, 2015.
- Huang et al. (2025)
Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Xun Zhou.
Over-tokenized transformer: Vocabulary is generally worth scaling.
*arXiv preprint arXiv: 2501.16975*, 2025.
URL [https://arxiv.org/abs/2501.16975](https://arxiv.org/abs/2501.16975).
- Huang et al. (2023)
Qian Huang, Eric Zelikman, Sarah Chen, Yuhuai Wu, Gregory Valiant, and Percy S Liang.
Lexinvariant language models.
*Advances in Neural Information Processing Systems*, 36:23990–24012, 2023.
- Jackendoff (2011)
Ray Jackendoff.
What is the human language faculty?: Two views.
*Language*, 87(3):586–624, 2011.
- Jozwik et al. (2017)
Kamila M Jozwik, Nikolaus Kriegeskorte, Katherine R Storrs, and Marieke Mur.
Deep convolutional neural networks outperform feature-based but not categorical models in explaining object similarity judgments.
*Frontiers in psychology*, 8:1726, 2017.
- Kallini et al. (2025)
Julie Kallini, Shikhar Murty, Christopher D Manning, Christopher Potts, and Róbert Csordás.
Mrt5: Dynamic token merging for efficient byte-level language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Kallmayer et al. (2020)
Aylin Kallmayer, Jacob Prince, and Talia Konkle.
Comparing representations that support object, scene, and face recognition using representational trajectory analysis.
*Journal of Vision*, 20(11):861–861, 2020.
- Katz et al. (2024)
Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf.
Backward lens: Projecting language model gradients into the vocabulary space.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 2390–2422, 2024.
- Kipper et al. (2006)
Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer.
Extending verbnet with novel verb classes.
In *Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)*, 2006.
- Kipper et al. (2008)
Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer.
A large-scale classification of english verbs.
*Language Resources and Evaluation*, 42:21–40, 2008.
- Korhonen & Briscoe (2004)
Anna Korhonen and Ted Briscoe.
Extended lexical-semantic classification of english verbs.
In *Proceedings of the Computational Lexical Semantics Workshop at HLT-NAACL 2004*, pp. 38–45, 2004.
- Kriegeskorte et al. (2008)
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini.
Representational similarity analysis-connecting the branches of systems neuroscience.
*Frontiers in systems neuroscience*, 2:249, 2008.
- Land & Bartolo (2024)
Sander Land and Max Bartolo.
Fishing for magikarp: Automatically detecting under-trained tokens in large language models.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 11631–11646, 2024.
- Lepori & McCoy (2020)
Michael Lepori and R Thomas McCoy.
Picking bert’s brain: Probing for linguistic dependencies in contextualized embeddings using representational similarity analysis.
In *Proceedings of the 28th International Conference on Computational Linguistics*, pp. 3637–3651, 2020.
- Levin (1993)
Beth Levin.
*English verb classes and alternations: A preliminary investigation*.
University of Chicago press, 1993.
- Liang et al. (2023)
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa.
Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 13142–13152, 2023.
- Mandelbrot (1953)
Benoit Mandelbrot.
An informational theory of the statistical structure of language.
*Communication theory*, 84:486–502, 1953.
URL [http://pdodds.w3.uvm.edu/research/papers/others/1953/mandelbrot1953a.pdf](http://pdodds.w3.uvm.edu/research/papers/others/1953/mandelbrot1953a.pdf).
- Mikolov et al. (2013)
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their compositionality.
*Advances in neural information processing systems*, 26, 2013.
- Nili et al. (2014)
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson, and Nikolaus Kriegeskorte.
A toolbox for representational similarity analysis.
*PLoS computational biology*, 10(4):e1003553, 2014.
- Nivre et al. (2020)
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman.
Universal dependencies v2: An evergrowing multilingual treebank collection.
In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pp. 4034–4043, 2020.
- Papadimitriou et al. (2022)
Isabel Papadimitriou, Richard Futrell, and Kyle Mahowald.
When classifying grammatical role, bert doesn’t care about word order… except when it matters.
In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pp. 636–643, 2022.
- Patil et al. (2022)
Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi.
Overlap-based vocabulary generation improves cross-lingual transfer among related languages.
In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 219–233, 2022.
- Pennington et al. (2014)
Jeffrey Pennington, Richard Socher, and Christopher D Manning.
Glove: Global vectors for word representation.
In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*, pp. 1532–1543, 2014.
- Piantadosi (2014)
Steven T Piantadosi.
Zipf’s word frequency law in natural language: A critical review and future directions.
*Psychonomic bulletin & review*, 21:1112–1130, 2014.
- Pinker (1998)
Steven Pinker.
Words and rules.
*Lingua*, 106(1-4):219–242, 1998.
- Press & Wolf (2017)
Ofir Press and Lior Wolf.
Using the output embedding to improve language models.
In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pp. 157–163, 2017.
- Pustejovsky (1998)
James Pustejovsky.
*The generative lexicon*.
MIT press, 1998.
- Raffel et al. (2020)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
*Journal of Machine Learning Research*, 21(140):1–67, 2020.
URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html).
- Schmidt et al. (2024)
Craig Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner.
Tokenization is more than compression.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 678–702, 2024.
- Silveira et al. (2014)
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning.
A gold standard dependency corpus for English.
In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)*, 2014.
- Timkey & van Schijndel (2021)
William Timkey and Marten van Schijndel.
All bark and no bite: Rogue dimensions in transformer language models obscure representational quality.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 4527–4546, 2021.
- Tirumala et al. (2022)
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan.
Memorization without overfitting: Analyzing the training dynamics of large language models.
*Advances in Neural Information Processing Systems*, 35:38274–38290, 2022.
- Tomasello (2005)
Michael Tomasello.
*Constructing a language: A usage-based theory of language acquisition*.
Harvard university press, 2005.
- Wies et al. (2021)
Noam Wies, Yoav Levine, Daniel Jannai, and Amnon Shashua.
Which transformer architecture fits my data? a vocabulary bottleneck in self-attention.
In *International Conference on Machine Learning*, pp. 11170–11181. PMLR, 2021.
- Wu et al. (2023)
Zhengxuan Wu, Alex Tamkin, and Isabel Papadimitriou.
Oolong: Investigating what makes transfer learning hard with controlled studies.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 3280–3289, 2023.
- Xue et al. (2022)
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel.
Byt5: Towards a token-free future with pre-trained byte-to-byte models.
*Transactions of the Association for Computational Linguistics*, 10:291–306, 2022.
- Ylonen (2022)
Tatu Ylonen.
Wiktextract: Wiktionary as machine-readable structured data, 2022.
URL [http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.140.pdf](http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.140.pdf).
Accessed 2025-04-26.
- Zhang et al. (2017)
Zhihao Zhang, Jennifer Fanning, Daniel B Ehrlich, Wenting Chen, Daeyeol Lee, and Ifat Levy.
Distributed neural representation of saliency controlled value and category during anticipation of rewards and punishments.
*Nature communications*, 8(1):1907, 2017.
- Zhelezniak et al. (2019)
Vitalii Zhelezniak, Aleksandar Savkov, April Shen, and Nils Y Hammerla.
Correlation coefficients and semantic textual similarity.
In *Proceedings of NAACL-HLT*, pp. 951–962, 2019.
- Zipf (1936)
George Kingsley Zipf.
*The Psychobiology Of Language*.
Routledge, 1936.
URL [https://books.google.com/books?id=oUr_AQAAQBAJ](https://books.google.com/books?id=oUr_AQAAQBAJ).

## Appendix A OLMo Results

Figure: (a) Semantic Measures
Refer to caption: /html/2510.07613/assets/x10.png

Figure: (a) Frequency convergence
Refer to caption: /html/2510.07613/assets/x12.png

## Appendix B Output Embeddings (Unembeddings) Results

Figure: (a) Semantic Measures
Refer to caption: /html/2510.07613/assets/x14.png

Figure: Figure 8: Pythia output embeddings results for Experiment 2
Refer to caption: /html/2510.07613/assets/x16.png

## Appendix C Experiment 1 with cosine and euclidean distances

As a sanity check for the effect of using Spearman distance, we re-ran the semantic experiments of Experiment 1 using cosine and Euclidean distances, and found that the results look almost identical ([Figure 9](#A3.F9)).

Figure: (a) [Figure 2(b)](#S4.F2.sf2), using cosine distance between token embeddings instead of Spearman
Refer to caption: /html/2510.07613/assets/x17.png

## Appendix D Similarity and Relatedness Splits

Figure: Figure 10: Model embeddings represent similarity much more than they represent relatedness. This is in contrast to our baseline of GloVe vectors (dotted black lines), which correlate with the Similarity and Relatedness splits much more similarly.
Refer to caption: /html/2510.07613/assets/x21.png

## Appendix E Regex for counting word frequency

To determine the frequency of each word, we counted the number of times every string that matched the following regex appeared in the C4 corpus (Raffel et al., 2020).

This captures:

- •
Numbers with commas and decimals
- •
Hyphenated words (well-known, know-it-all), excluding sequences longer than 3 words (which are often websites etc)
- •
Acronyms (U.S.A., Ph.D)
- •
Words with apostrophes (don’t, can’t)
- •
Standard words

## Appendix F Convergence RSA for syntactic categories

Figure: (a)
Refer to caption: /html/2510.07613/assets/x22.png

#### Function words converge earlier in training than lexical words

In [Figure 11(b)](#A6.F11.sf2), we present our results for the convergence of different parts of speech. We report the average convergence for the two primary classes of parts of speech: functional parts of speech (pronouns, adpositions, auxiliaries, conjunctions, determiners, numerals, particles, and punctuation) and lexical parts of speech (nouns, verbs, proper nouns, adjectives, and adverbs). This differentiates words that have a largely grammatical use to those that usually impart lexical meaning to sentences. Our results show that functional words, which serve a more grammatical purpose, converge to their final representational space earlier in training than lexical words.

#### Lexical word representations are correlated with their random initializations

Lastly, it is notable to examine the very beginning of our syntactic convergence RSA plots, at training step 0, which we reproduce in more detail in the inset in [Figure 11(b)](#A6.F11.sf2). These values show how correlated the final relationships represented in the model embeddings are to the random initialization of the embedding matrix. Interestingly, this correlation is not zero for either of the two classes, and is even higher than 0.1 for the lexical classes. This indicates that the random initialization influences how models represent the relationships between words, and that this random influence cannot be overcome even with a lot of training.

## Appendix G Full results: Words that move most to be closer and farther from each other after the first 20,000 training steps:

Figure: Figure 12: Top word pairs that get closer between 20,000 training steps and 142,000 training steps. The vast majority of the words that get closer are inflectional forms of rare, technical words. The maximum change in correlation for any two pairs getting closer is 0.47

Figure: Figure 13: The top word pairs that get farther between 20,000 training steps and 142,000 training steps. The majority of these pairs are function words with unintelligble or word fragments.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1) If I understood correctly, convergence RSA compares the model's representation at different training stages. While RSA seems suited for comparing model representations to human annotated ressources, is it equally appropriate when the representational system is the same? Can other comparisons could be considered in that case?

2) Given the relatively small number of word pairs in Experiment 1, how representative is this subset of the vocabulary, and could this affect the reported trends?
soundness: 4
strengths: 1) The paper provides a handful of experiments to characterize vocabulary representation across several dimensions. The experiments and claims are clearly explained. I found the combination of hypothesis RSA and convergence RSA especially interesting. 

2) This methodology could be adapted to other languages and very interesting, especially for language with structures highly different from English, though this would require significant additional resources.
summary: The paper is a descriptive study that investigates how vocabulary representations of language models evolve during training. The experiments focus on two English language models (Pythia-12B and OLMo-7B) for which several training checkpoints are available, allowing to study their evolution over time. 

Across three experiments, the paper studies the relationship between the vocabulary representations at each training checkpoint and 1) human annotated datasets for vocabulary structure (semantic and syntactic similarity), 2) word frequency in the training corpus and 3) the final representation of the fully trained model. 

The paper uses Representational Similarity Analysis (RSA) to compare the proximity between two different representational spaces, model's representations and the human's annotation (or word frequency). 

The study finds that the models' representations of wordss correlate most strongly with human semantic and syntactic annotations early in training (around 15% of total steps), that representations of frequent words converge earlier than those of less frequent words, but also that nevertheless, the embeddings continue to change during the remainder of training, especially for rare and technical tokens. 

The paper claims that these findings could help to better understand the training mechanisms of language models.
weaknesses: 1) The paper studies two models, but doesn't and explicitely states that those models are in English and that the claims are only supported for that language. I also think that more models might be necessary to confirm the claims. The paper would benefit from quickly clarifying their architectures, and the content of their training data. 

2) Figure are sometimes distant from their discussion in the text (figure 2, and figures in the appendix). The main figures appear to combine both models to show general trends, while the details for each model are in appendix. 
To me at least some of the per-model results contain key information that should appear in the main body. Having to switch repeatedly between the text and appendix made the results harder to follow. 

3) The experiments are thorough, but some findings (faster convergence of frequent tokens) are largely intuitive, which limits the novelty of the contribution

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: * It is not clear how the authors perform the hypothesis RSA. According to my understanding, the task aims to assess the correlation between embedding‐based distances and human-annotated labels, but each step in this process is not clearly stated. It would be useful to introduce some notations to describe the problem, input, output and evaluation protocol. I have difficulties understanding this RSA, especially for the syntactic analysis. 
* In line 278 *"and if a word appears as a part of speech at least 5 times it receives distance 0 to all other words with that part of speech"*, I do not fully understand what is the meaning of this sentence
soundness: 2
strengths: * This work explores how does he embedding layer in transformers learn knowledge durting the training stage, which is interesting and novel
* Their findings track the representative changes of tokens and reveal their connections between initial random embeddings and the learned embeddings. The study of the less-frequent tokens might be useful for future studies since LLMs tend to fail on long-tail and domain-specific cases.
summary: This work uses representational similarity analysis (RSA) to examine how vocabulary embeddings in language models evolve during the training phase. Basically, they conduct experiments to study the correlation between embedding-based distance and human-annotated distance for several tasks: semantic similarity, syntactic similarity, and word frequency. Their main findings indicate that 
* The semantic and syntactic converge to high correlations at an early training step
* Embeddings of high-frequency and function words converge to their final vectors faster than lexical and low-frequency words
weaknesses: * I appreciate that the authors aim to answer this question: what are the representative changes of word embeddings during training, and this work provides some findings. However, I would expect some more insightful analyses. (1) LLMs are known for their contextual capabilities, which means they understand each word according to the context surrounding it rather than using the static embedding layer. What is the motivation to study this static embedding matrices? Can we apply the main findings in this manuscript to improve the current architecture or training strategies? (2) the authors found that the semantic and syntactic converge to high correlations at an early training step. Why do the embeddings exhibit such a pattern?  It would be more helpful if the authors could go further and offer explanation for their observations. (3)  The second main finding *"Embeddings of high-frequency and function words converge to their final vectors faster than lexical and low-frequency words"* is aligned with our intuition, which is not surprising. I would suggest exploring this point: one known issue of LLMs is that they tend to generate hallucinated answers for long-tail queries. Is it possible to learn the correlation between hallucination and low-frequency words? If so, it would be very useful to find solutions to mitigate this issue.
* The presentation requires another round of revision. I have difficulties understanding the details of the methodology and I have to guess to complete this review. I would suggest adding basic notations for clarification.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: Q1: What do the reported RSA patterns (your checkpoint-wise correlations between embedding distances and linguistic hypotheses) tell us about model quality in practice? Do the early emergence or early peak you highlight correspond to any observable improvement in the model’s ability to predict text or handle simple behaviors?

Q2: Please clarify the contribution to interpretability beyond descriptive timelines. Do these observations translate into explicit actions in practice, and if so, which training or evaluation decisions should reasonably be guided by them?

Q3: In Section 4, how do you distinguish structure that arises because certain words appear far more often during training from structure that reflects genuine syntactic or semantic organization?

Q4: In Section 5, how does the relationship between frequency groups evolve (e.g., head vs. tail as a whole)? Do cross-group relationships stabilize earlier, later, or differently than within-group neighborhoods?

Q5: In Section 5, why is the last checkpoint an appropriate reference for your "convergence" analyses? How sensitive are your conclusions to that choice, and are low-frequency words actually stable by the end of training?

Q6: Several frequency buckets appear out of order in figures (e.g., a lower-frequency curve above a higher-frequency one). How should readers reconcile this with statements that high-frequency words show "substantially stronger" effects?

Q7: Related to Q6, what level of variability or uncertainty is present in your curves and bucket comparisons? Can you clarify how confident we should be about the ordering across frequency groups?
soundness: 2
strengths: This study has a strong empirical scope as it assembles dense checkpoint trajectories across two modern model families. The applied RSA technique is principled, and the reported early semantic plateau is clear and replicated across models. The frequency-aware slicing idea and the comparison to the final checkpoint are interesting.
summary: This paper studies how and when the (input) vocabulary embeddings of Large Language Models (LLMs) organize themselves to reflect linguistic properties. Using Representational Similarity Analysis (RSA) on the Pythia 12B and OLMo 7B models, the authors track the pairwise distances of the input embeddings throughout the training process. They find that this linguistic structure emerges very early in training, since the embedding space quickly converges to a state that shows high correlation with both semantic features and syntactic features. They also find that word frequency plays a major role as high-frequency words converge to their final representations much faster than low-frequency words.
weaknesses: **W1. Operational significance is unclear.**

The paper reports Hypothesis-Driven RSA and Convergence RSA curves, but it does not show how their curves relate to model perplexity or to any simple downstream behavior. Because perplexity is the training objective, it is important to see whether the reported “early emergence” (for semantics) or “early peak” (for syntax) coincides with changes in perplexity. Without such a link, the reader cannot tell whether these RSA events matter for model quality or are just descriptive patterns.

**W2. Frequency is a likely confound that is not controlled.**

The paper reports "early" emergence for both semantic and syntactic RSA (Section 4), but it never controls for token exposure. Frequent words appear many more times and therefore get many more updates; this alone can create an "early" effect. Section 5 acknowledges the role of frequency by slicing into buckets, yet it does not revisit the "syntax" claim after holding frequency fixed. As written, "early syntax" can simply be a frequency artifact.
To resolve this concern, the authors should modify their Experiment 1 to build matched sets with controlled frequency distributions and recompute the RSA curves. 

**W3. Cross-bucket (global) structure is not measured.** 

Section 5 looks only within each bucket (100×100 distance matrices per bucket). It never examines how different buckets sit relative to each other over time (for example, whether the center of the top-100 words moves closer to or farther from the center of the 900–1000 group). It is entirely possible that the global arrangement across buckets stabilizes early while the finer within-bucket neighborhoods keep shuffling. If so, the paper's timing claims would be incomplete.

**W4. "Convergence to final" is fragile and can mislead**

Convergence RSA defines "stability" as "becoming similar to the final checkpoint." This assumes the last snapshot is the right anchor, even though late steps can have extra noise or idiosyncrasies. As a result, a subset can look "stable" simply because it mirrors quirks of the final checkpoint. This matters because the paper's main takeaways about "who stabilizes when" depend on this anchor. It also affects the interpretation in Section 5: when time is rescaled by expected exposures, the narrative ("low-frequency words converge quickly per exposure") competes with another observation in the paper, i.e., low-frequency words keep stronger ties to initialization. If those words have not actually stabilized by the final checkpoint, "convergence to final" is not a reliable guide for them.

**W5. Claims and presentation need correction and calibration.**

The paper uses ambiguous terminology and legends. Two examples are given below: 

- In Section 6, the text alternates among "distance decreased," "embeddings moved closer," and "similarity increased" to describe the same pairwise change. This slows the reader and risks confusion.

- In Figure 3(a), the legend says "Frequency bucket upper limit (100 words/bucket)," but the numbers are frequency ranks (larger number = less frequent), so the correct label is "Frequency rank bucket (100 words/bucket)".

Furthermore, some of the statements in the paper seem overconfident. In Figure 4(b), the text states that "high-frequency words (light blue) show substantially stronger correlations between input and output embeddings than low-frequency words," yet several lower-frequency curves sit above higher-frequency ones at many points, and no uncertainty bands are shown. The reasonable claim here is "on average" rather than an unconditional statement.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: How could "the methodological choice of a large token embedding matrix in LLMs implicitly build in the assumption of a words-and-rules approach to language in the system?"

Virtually all kinds of structure you analyse saturate in the first half of pre-training. Are you claiming no further linguistic information is learned later on?

Does when "linguistic features" stabilise relate to performance in any way? Olmo has a higher peak, is it better at a particular task?
soundness: 3
strengths: This paper is clearly written, giving the reader a clear sense of their methods and results. The information is well presented and largely well scoped to the scale of a conference paper. The results related to frequency information in particular are interesting, shedding some light on how frequency information affects the training process.
summary: This paper studies how the embedding matrix of an LLM evolves over the course of pre-training. Leveraging two variants of representational similarity analysis this work looks at how linguistic features, and frequency information develop in the Olmo and Pythia families of models. Results show linguistic information converges early on, representations for high frequency words also converge early in training with low frequency words taking much longer. Low frequency words also retain similarity structure with their random initialisation more than their high frequency counterparts.
weaknesses: The two main weaknesses appear to be a limited degree of novelty and a lack of theoretical clarity in the relationships the authors draw to linguistics. In terms of novelty, when during training syntactic information is represented in an LLM is well studied, to a lesser degree effects of frequency information are as well. The authors make clear that much of this work has studied model activations rather than the input embedding matrix. While this may be true the overwhelming majority of computation in a model happens inside of the model. As a result though there is novelty here it seems quite narrow.

Of the results, the results showing lower frequency tokens retain similarity structure with their random initialisation are particularly interesting. However the results in these sections are largely a list of empirical findings with limited analysis or explanatory account - instead offering claims like "frequency [is] a primary factor in driving embedding changes" (a fact established in previous work), the question is how frequency drives those changes. The authors have some results to this effect, it would be great to see a clearer line of argumentation of how frequency affects training.

The stated relationship between this work and linguistics is difficult to follow. Looking particularly at the "Structure in the vocabulary" paragraph (line 112). The "words and rules" approach attributed in Pinker is in fact the broader Generative Tradition in linguistics, which has historically emphasised the separation of syntactic and semantic processes. These are contrasted by Usage-Based approaches to language which argue syntax and semantics are broadly inseparable, with semantic constructions also being a part of the generating process. As written this paper describes this as the "syntax-lexicon" continuum --- a lexicon is not the same as semantics. All accounts cited broadly agree with the existence of syntax and a lexicon, but disagree about what is in the lexicon - this is the line of argumentation that aligns with the authors experiments but as a reader is not the argument they appeared to make. For reference there's a particular disagreement between Jackendoff 1990 and Goldberg 1995 (which they cite) about the degree to which syntactic information is present in the lexicon.

An issue with drawing comparisons with Usage-Based accounts is that they emphasise the importance of context in conditioning and driving the generative process, the author's choice to consider only the embedding matrix makes it hard to directly relate the results here on non-contextualised embeddings to this work in linguistics. Additionally in Usage-Based accounts we would expect semantic, syntactic, and frequency information to be entangled. The analysis here isolates proxies for these kinds of information in separate labels rather than in combination - the authors note this point at 348 that embeddings may be "better explained by non-trivial combinations of linguistic features".

In the conclusion the authors state: "Does the methodological choice of a large token embedding matrix in LLMs implicitly build in the assumption of a words-and-rules approach to language in the system?" I struggle to understand what this question is asking. In the human case, we take raw words, or phonemes as inputs yet may process them as contextualised semantic concepts like constructions. How could having a discrete input, or an embedding matrix for your discrete input entail a factorisation of syntax and semantics?

To be clear I think the methods here are sound, and some of the results are interesting. However the authors emphasise the relationship between their work and related work in linguistics, in part to add theoretical novelty to their use of only the embedding matrix. As a result their engagement with linguistic theory needs to be accurate, and well described.


Decision: Reject