Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training
Authors: Max Müller-Eberstein
Rob van der Goot
Barbara Plank
Ivan Titov
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Modelling
Representation Learning
Learning Dynamics
Probing
Subspace Analysis
TL;DR: Leveraging a novel probing suite for comparing representational embedding subspaces across tasks and time, this work identifies critical learning phases during language model training in which subspaces emerge, share information, and specialize.

Abstract: Representational spaces learned via language modeling are fundamental to Natural Language Processing (NLP), however there has been limited understanding regarding how and when during training various types of linguistic information emerge and interact. Leveraging a novel information theoretic probing suite, which enables direct comparisons of not just task performance, but their representational subspaces, we analyze nine tasks covering syntax, semantics and reasoning, across 2M pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize. Across these phases, syntactic knowledge is acquired rapidly after 0.5% of full training. Continued performance improvements primarily stem from the acquisition of open-domain knowledge, while semantics and reasoning tasks benefit from later boosts to long-range contextualization and higher specialization. Measuring cross-task similarity further reveals that linguistically related tasks share information throughout training, and do so more during the critical phase of learning than before or after. Our findings have implications for model interpretability, multi-task learning, and learning from limited data.

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training
Max MbileenEberrtein Rob van der Goot Barbara Plank @A Ivan Titoy We
Department of Computer Science, IT University of Copenhagen, Denmark
Center for Information and Language Processing (CIS), LMU Munich, Germany
Munich Center for Machine Learning (MCML), Munich, Germany ILCC, University of Edinburgh, United Kingdom ILLC, University of Amsterdam, The Netherlands
mamy@i tu. tu.dk, robv@itu.dk, b. b.plankelmu.c de, ititovêinf.en ed. ac.
Abstract
Representational spaces learned via language modeling are fundamental to Natural Language Processing (NLP), however there has been lim- ited understanding regarding how and when during training various types of linguistic infor- mation emerge and interact. Leveraging novel information theoretic probing suite, which en- ables direct comparisons not just task perfor- mance, but their representational subspaces, analyze nine tasks covering syntax, semantics and reasoning, across 2M pre- pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentan- gle to specialize. Across these phases, syntac- tic knowledge is acquired rapidly after 0.5% of full training. Continued performance im- provements primarily stem from the acquisition open-domain knowledge, while semantics and reasoning tasks benefit from later boosts to long-range contextualization and higher spe- cialization. Mcasuring cross-task similarity further reveals that linguistically related tasks share information throughout training, and do more during the critical phase of learning than before or after. Our findings have impli- cations for model interpretability, multi-task learning, and learning from limited data.
Tasls
VEL
Taalo
LM,
LM
LMz
pro-train Corpus
Time
Figure I: Subspace Chronicles via probes 0 across LM training time, measured by F1, codelength L, and subspace angles across tasks and time.
LM training objective begins to encode informa- tion that is relevant to downstream tasks is crucial, as this informs the limits of what can be Iearned using current objectives.
For identifying task-relevant information in LMs, probing has become an important tool (Adi et al., 2017; Conneau et al., 2018; Giulianelli et al., 2018; Rogers et al., 2020), which has already shown promise for revealing LM learning dynam- ics (Saphra and Lopez, 2019; Chiang al., 2020; Liu et al., 2021). The predominant approach of quantifying linguistic information via task perfor- mance, however, misses important interactions at the representational level, i.e., whether actual lin- guistic information is present at any given training timestep (Hewitt and Liang, 2019; Voita and Titov, 2020), and how these representational subspaces change independently of model performance. Fur- thermore, performance alone does not indicate how different tasks and their related linguistic informa- tion interact with each other during training. and how much information they actually share.
1 Introduction
Contemporary advances in NLP are built on the representational power of latent embedding spaces learned by self-supervised language models (LMs). At their core, these approaches are built on the dis- tributional hypothesis (Harris, 1954; Firth, 1957), for which the effects of scale have been implic- itly and explicitly studied via the community's use of increasingly large models and datasets (Teehan al., 2022; Wei al., 2022). The learning dy- namies by which these capabilities emerge during LM pre-tnaining have, however, remained largely understudied. Understanding how and when the
By leveraging cntrmatiin-thcoruae probes as characterizations of task-specific subspaces within

---

LM's overall embedding space (Figure 1), we aim to answer these questions, and contribute:
2020) and RoBERTa (Zhuang al., 2021) respec- tively. They similarly identify that performance on syntactic tasks increases faster than on world knowledge or reasoning tasks, and that this pattern holds across pre-training corpora from different domains. This points towards consistent LM learn- ing dynamics. however using only performance or correlations, it is difficult to interpret how the rep- resentation of this knowledge changes over time well as how it overlaps across tasks.
An information theoretic probing suite for ex- tracting not just performance, but entire task- specific representational subspaces within an LM's embedding space, allowing us to mea- sure changes to linguistic information over time and across tasks (Section 3).
A study of task subspace emergence, shifts and interactions across nine diverse linguistic tasks, 2M pre-trate training steps and five random initializations (Section 5).
Similar issues have arisen in single-checte checkkonn probing where seemingly task-specific information is identified even in random models, requiring con- trol tasks (Hewitt and Liang, 2019), and causal interventions, such as masking the information quired solve a task (Lasri et al., 2022; Hanna et al., 2023). By using the rate of model's com- pression of linguistic information, Voita and Titov (2020) alternatively propose an information the- oretic measure to quantify the consistency with which task-relevant knowledge is encoded.
An analysis of these learning dynamics, which focuses on practical implications beyond per- formance to identify what can be learned given limited data, and how to effectively leverage this knowledge (Section 6).
2 Related Work
Another limitation of prior learning dynamics work is the inability to compare similarities across fasks-ic., whether information is being shared. Aligned datasets, where the same input anno- tated with different labels, could be used to mea- sure performance differences across tasks in a more controlled manner, however they are only available for limited set of tasks, and do not provide a direct, quantitative measure of task similarity. By combining advances in information-theone probing with measures for subspace geometry, we propose a probing suite aimed at quantifying task- specific linguistic information, which allows for subspace comparisons across time as well as across tasks, without the need for aligned datasets.
Although prior work has not specifically focused on the emergence of representational spaces, but rather on task performance, there has been an increased interest in the emergent capabilities of LMs. The community-wide trend of increasing model and dataset size has shown that certain skills (e.g., arith- metic) are linked to scale (Wei al., 2022; Scha- effer et al., 2023), however Teehan et al. (2022) si- multaneously identify a lack of work investigating skill emergence across the training time dimension.
To promote research in this direction, some projects have released intermediate training check- points. This notably includes MultiBERTs (Sellam et al., 2022) which studies cross-inttaalization vari- ation BERT (Devlin al., 2019), as well as Mis- tral (Mistral, 2022), Pythia (Biderman et al., 2023) and BLOOM (Scao et al., 2023) which release checkpoints across training steps, seeds and/or pre- processing procedures.
Emergent Subspaces
To compare linguistic subspaces over time, we re- quire snapshots of an LM across its training process (Section 3.1), a well-grounded probing procedure (Section 3.2), and measures for comparing the re- sulting subspaces (Section 3.3).
In seminal work investigating learning dynamics, Saphra and Lopez (2019) measure how strongly hidden representations of a BiL.SMM LM corre- late with parts-of-speech (PoS), semantic tags and topic information. By probing these characteristics across LM training time, they identify that PoS is acquired earlier than topics. For Transformer mod- els. Chiang al. (2020) and Liu et al. (2021) probe intermediate checkpoints of ALBERT (Lan et al.,
3.1 Encoders Across Time
For intermediate LM checkpoints, make use of the 145 models published by Sellam et al. (2022) as part of the MultBERT's project. They not only cover 2M training deps-double that of the origi- nal BERT (Devlin et al., 2019) but also cover five seeds, allowing us to verify findings across multiple
Code at http::/ghhh...cmmmmii//aaaccccco

---

initializations. The earliest available trained check- point is at 20k steps. Since our initial experiments showed that performance on many tasks had al- ready stabilized at this point, we additionally train and release our own, more granular checkpoints for the early training stages between step 0-20k, for which we ensure alignment with the later, official checkpoints (details in Section 4.1).
the probe 0 itself. The variational formulation of information-theorceee probing, which we use our work, measures codelength by the bits-back com- pression algorithm (Honkela and Valpola, 2004). It is given by the evidence lower-bound:
L --
Σ F.ED
log2 Po (vl
(I)
3.2 Probing for Subspaces
(L((1)))
Shifting our perspective from using probe to mea- sure task performance the probe itself repre- senting task-specific subspace is key to enabling cross-task comparisons. In essence, a probe char- acterizes a subspace within which task-relevant information is particularly salient. For extract- ing c-dimensional subspace (with c being the number of classes in a linguistic task) from the d-dimensional general LM embedding space, we propose modified linear probe 0 E Raxe which learns to interpolate representations across the I LM layers using a layer weighting C RR (Tenney al., 2019). By training the probe to classify the data (x, v) E DD the resulting / thus corresponds to the linear subspace within the overall LM which maximizes task-relevant information.
where the cost of transmitting the probe corre- sponds to the Bellbackk-eibler divergence between the posterior distribution of the probe parameters ~ 3 and a prior 7, The KL divergence term quantifies both the complexity of the probe with respect to the prior, as well as regularizes the train- ing process towards probe which achieves high performance, while being as close the prior possible. Following Voita and Titov (2020), we use sparsity-inducing prior for y, such that the result- ing is as small of a transformation as possible. In contrast to measuring task information via per- formance alone, codelength L allows us to detect how consistently linguistie information is encoded by the LM (i.e., the shorter the better). At the same time, the linear transformation 0 becomes an efficient characterization of the uask-specific repre- sentational subspace. To further ground the amount of learned information against random representa- tions, we use the probes of the randomly initialized models at checkpoint 0 as control values.
To measure the presence of task-specific infor- mation encoded by an LM, it is common to use the accuracy of probe predicting a related lin- guistic property. However, accuracy fails to cap- ture the amount of effort required by the probe to map the original embeddings into the task's 's label space and achieve the relevant level of performance. Intuitively, representations containing consistent and salient task information are easier group by their class, resulting in high performance while re- quiring low probe complexity (and/or less training data). Mapping random representations with no class-wise consistency to their labels on the other hand requires more effort, resulting in higher probe complexity and requiring more data.
3.3 Subspace Comparisons
As each probe @ characterizes task-specific sub- space within the overall embedding space, compar- isons between subspaces correspond to measuring the amount of information relevant both tasks. To ensure that the subspaces extracted by our prob- ing procedure are fully geometrically comparable, they are deliberately linear. Nonetheless, multiple factors must be considered to ensure accurate com- parisons: First, matrices of the same dimensionality may have different rank, should one subspace be casier to encode than another. Similarly, one ma- trix may simply be a scaled rotated version of another. Correlating representations using, e.g., Singular Vector or Projection Weighted Canonical Correlation Analysis (Raghu et al., 2017; : Morcos et al., 2018) further assumes the underlying inputs
Information-theoreei probing (Voita and Titov, 2020) quantifies this intuition by incorporating the notion of probe complexity into the measure of task-relevant information. This is achieved by re- casting the problem of training probing classifier learning transmit all (xr, v) 'D in as few bits as possible. In other words, it replaces probe accu- racy with codelength I., which is the combination the probe's quality of fit to the data as measured by ()(y) over D. and the cost of transmitting

---

be the same such that only the effect of repre- sentational changes measured. When comparing across datasets with different inputs a and different labels y, this is no longer given.
BookCorpus (Zhu et al., 2015). As the exact data are unavailable, Sellam al. (2022) use an alter- native corpus by Ture et al. (2019), which aims to reproduce the original pre-training data. Unfor- tunately, the latter is also not publicly available. Therefore, we gather similar corpus using fully public versions of both corpora from the Hugging- Face Dataset Hub (Lhoest et al., 2021). We further provide scripts to re-create the exact data ordering, sentence pairing and subword masking to ensure that both our LMs and future work can use the same data instances in exactly the same order. Note how- ever that our new checkpoints will have observed similar, but slightly different data (Appendix A. A.1..
To fulfill the above requirements, we make use of Principal Subspace Angles (SSAs; Knyazev and Argentati, 2002). The measure is closely related to Grassmann distance (Hamm and Lee, 2008) which has been used to, e.g., compare low-rank adaptation matrices (Hu et al., 2022). It allows us to compare sask-specific subspaces 9 in their entirety and inde- pendently of individual instances, removing the re- quirement of matching x across tasks. The distance between two subspaces 0AA E Raxp and 0B E Rtdx intuitively corresponds to the amount of 'eneryy' re- quired to map one to the other. Using the orthonor- mal bases Q = orth(f and QB orth(0;) of each subspace to compute the transformation mag- nitudes M = QBQB furthermore ensures linear in- variance. The final distance is obtained by convert- ing M's ss singular values UVV' = SVD(M) into angles between 0 and 90º (i.e., similar/dissimilar):::
Training Given the generated data order, the new LMs are trained according to Sellam et al. (2022), using the masked language modeling (MLM) and next sentence prediction (NSP) objectives. One up- date step consists of batch with 256 sentence pairs for NSP which have been shuffled to be 50% cor- rect/incorrect, and within which 15% of subword tokens (but 80 at most) are masked or replaced. For the optimizer, learning rate schedule and dropout, we match the hyperparameters of the original work (details in Appendix B. Even across five ini- tializations, the environmental impact of these LM training runs is low, as re-use the majority of checkpoints from MultiBERTs beyond step 20k.
(S, = arccos(diag(=)).
(2)
We use SSAs to quantify the similarity between subspaces of the same task across time, as well as to compare subspaces of different tasks.
4 Experiment Setup
4.1 Early Pre-training
Pre-training Results For LM training, we ob- serve a consistent decrease in MLM and NSP loss (see Appendix C. As our models and MultiB- ERTs are not trained on the exact same data, we find that SSAs between our models and the origi- nals are around 60%, which is within the bounds of the angles we observe within the same training run in Figure 4 of Section 5.2. This has no effect on our later analyses which rely on checkpoints from within a training cohort. Surprisingly, we further find that although these models start from the same initialization, but are trained on different data, they are actually more similar to each other than models trained on the same data, but with different seeds. SSAs for checkpoints from the same timestep and training run, but across different seeds consistently measure >80 (Figure 13 in Appendix C.1), high- lighting that initial conditions have a stronger effect on representation learning than minor differences in the training data. Finally, our experiments in Section show that our reproduced checkpoints align remarkably well with the official checkpoints,
Model MultiBERT's (Sellam al., 2022) cover 2M training steps, however our initial experiments showed that the earliest model step 20k already contains closc-to-final amounts of information for many tasks (see Section 5). This was even more pronounced for the early checkpoints of the larger LMs mentioned in Section 2. As such we train own early checkpoints starting from the five MultiBERTs initializations at step O. By saving 29 additional checkpoints up to step 20k, we aim to analyze when critical knowledge begins to be acquired during early training. To verify that trajec- tories match those of the official checkpoints, we train up to step 40k and compare results. Further- more, we measure whether the subspace angles be- tween the original and additional models are within the bounds expected for models sharing the same random initialization.
BERT (Devlin et al., 2019) reportedly trained on 2.5B tokens from English Wikipedia (Wikimedia, 2022) and 800M tokens from the
Data

---

100
100
R65
P65
Dep
Dep
(macrr)
Codolenpth
60
60
Topk
Topic
Send
Sendi
DA
QA
20
NJ
10
10'
107
10
207
10a
10
10P
10a
10
Steps
Steps
Figure F1 (macro) LM Training Time on each task's dev split (standard deviation across seeds). Dark/light shaded areas indicate 95% 95%/90% maximum performance. Reproduced checkpoints until 19k, Multi- BERTs from 20k to 2M steps.
Figure Codelength Ratio over LM Training Time as percentage of bits required to encode model and data with respect random model (standard deviation across seeds). Lower codelength corresponds to higher compression and more task-relevant information.
and observe continuation of this trajectory for the overlapping models between steps 20k and 40k.
5 Results
Across the 2,610 probing runs, we analyze sub- space emergence (Section 5.1), shifts (Section 5.2), and their interactions across tasks (Section 5.3). the following figures, results are split between the official MultiBERT's and our own early pre-training checkpoints, which, as mentioned in Section 4.1, follow an overlapping and consistent trajectory, and have high representational similarity. Standard de- viations are reported across random initializations, where we generally observe only minor differences in terms of performance.
4.2 Probing Suite
Given the 29 original MultiERTts in addition to our own 29 early checkpoints, both covering five initializations, analyze total of 290 models using the methodology from Section 3. In order to cover broad range of tasks across the linguistic hierarchy, we extract subspaces from nine datasets analyzing the following characteristics: parts-of- speech (PoS), named entities (NER) and corefer- ence (COREF) from OntoNotes 5.0 (Pradhan et al.. 2013), syntactic dependencies (DEP) from the En- glish Web Treebank (Silveira et al., 2014), seman- tie tags (SEM) from the Parallel Mcaning Bank (Abzianidze et al., 2017), TOPIC from the 20 News- groups corpus (Lang, 1995), sentiment (SENTI) from the binarized Stanford Sentiment Treebank (Socher et al., 2013), extractive question answer- ing (QA) from the Stanford Question Answering Dataset (Rajpurkar et al., 2016), and natural lan- guage inference (NLI) from the Stanford Natural Language Inference Dataset (Bowman et al., 2015). Each task is probed and evaluated the token-level (Saphra and Lopez, 2019) to measure the amount of task-relevant information within each contex- tualized embedding. In Appendix A.2 we further provide dataset and pre- pree-roccssing details.
5.1 Subspace Emergence
Starting from macro-F1 performance, Figure 2 shows clear learning phases with steep increase between 1k and 10k update steps, followed by shal- lower growth until the end. Within subsets of tasks, we observe subtle differences: PoS and TOPIC ap- pear to converge (i.e., >90% of final F1) within the 10k range. SEM follows a similar pattern, but has higher variance due to its many smaller classes. DEP and COREF, also gain most in the 1k-10k phase, but continue slowly climb until converg- ing later around 100k steps. NER shares this slow incline and sees a continued increase even ter 100k steps. SENTI and NLI also have early growth and see another small boost after 100k steps which the other tasks do not. Finally, QA also improves after 100k steps, however results mostly around the 50% random bascline, the linear probe is unable to solve this more com- plex task accurately. These results already suggest task groupings with different learning dynamics, however with Fl alone, it is difficult to understand interactions of the underlying linguistic knowledge.
For each task, we train an information theoretic probe for maximum of 30 epochs using the hyper- parameters from Voita and Titov (2020). This re- sults in 2,610 total probing runs for which we mea- sure subword-level macro-IFl, and that, more im- portantly, yield task subspaces for which we mea- sure codelength, layer weights, and SSAs across tasks and time (setup details Appendix B.2).

---

90
12
Rd5
P65
Dep
Center of Grevily
Dep
55
Tppi
Topic
Send
Send
D4
QA
NU
NJ
10 10
10
10
100
10a
10a
10a
10a
107
10
Steps
Steps
Figure 4: Siep-wise SSAs between Probes indicating the degree of subspace change per update (standard deviation across seeds). Subspaces between original and reproduced checkpoints at step 20k not comparable.
Figure 5: Center of Gravity
Layers measured following Tenney et al. (2019), across LM training time (standard deviation across seeds). For detailed weight- ings of all layers, refer to Appendix C.2.
Furthermore, even the random models at step reach non-trivial scores (e.g.. PoS and COREF with >60% F1), as probes likely memorize random, but persistent, non-comtextualize embeddings simi- larly to a majority baseline. This highlights the challenge of isolating task-speciff knowledge us- ing performance alone.
here to it, exhibiting larger differences while the learning rate is low and vice versa. Figure 13 Appendix C additionally shows how SSAs across checkpoints at the same timestep, but across dif- ferent random seeds are consistently greater than 80 indicating high dissimilarity. Compared to models starting from the same initialization, even if trained on slightly different data (i.e., reproduced and original), these angles indicate task subspaces' higher sensitivity to initial representational condi- tions than to training data differences.
Turning to codelength in Figure 3, we mea- sure the actual amount of learned information as grounded by the level of compression with respect to the random intialization. This measure confirms the dynamics from the performance graphs, how- ever subspaces also continue changing to a larger degree than their performance counterparts sug- gest. Even after the 10k step critical learning phase PoS information continues be compressed more efficiently despite Fl convergence. In addition, codelength provides more nuance for COREF, QA and NLI, which see little performance gains, but for which subspaces are continuously changing and improving in terms of compression rate.
Another important dimension for subspaces in Transformer-based LMs is layer depth. We plot layer weighting across time in Figure 5 using the center of gravity (CoG) measure from Tenney et al. (2019), defined as Σ agi It summarizes the depth which the probe finds the most salient amounts of task-relevant information. Appendix C.2 further shows the full layer-wier wise weights across time in greater detail. These weightings shed fur- ther light onto the underlying subspace shifts which surface in the other measures. Note that while SSAs across random initializations were large, vari- ability of CoG is low. This indicates that the layer depth at which task information is most salient can be consistent, while the way is represented within the layers can vary substantially.
5.2 Representational Shifts
Both performance and codelength indicate distinct learning phases well continual changes to lin- guistic subspaces. Figure 4 plots these shifts by measuring subspace shifts as a function of SSAs from one timestep to the next. We observe particu- larly large updates after the first 100 steps, followed by smaller changes until the beginning of the steep learning phase at around Ik steps. During the large improvements between 1 and 10k steps, subspaces shift at steady rate of around 45 followed by decrease 5-20" the end of training. Once again, subspaces are shifting across the entire train- ing process despite Fl suggesting otherwise. Note that while learning rate scheduling can have effects on the degree of change, SSAs do not strictly ad-
CoG at step essentially corresponds to the contextualization level of a task; memorizing non- contextualized embeddings at layer 0 or leveraging some random, but consistent mixing in the later layers. At the start of learning between 100 and 1k steps, all task subspaces dip into the lower half of the model. Together with the previously observed high initial subspace shift, this indicates that, begin- ning from the non non-contextual embeddings in layer O, contextualization becomes increasingly useful

---

10a
10P
x
Figure 6: Subspace A Angles across Tasks start, end and each order of magnitude of LM training time. Large angles (darker) and small angles (lighter) correspond to low and high similarity respectively.
from the bottom up. Paralleling the steep improve- ments of the other measures, CoG similarly climbs up throughout the model until the end of the crit- ical learning phase in the 10k range. Until 500k steps, the layers for different tasks disentangle and stabilize. At this point syntactic and lower-level semantic tasks converge towards the middle lay- ers. This specialization is especially prominent for TOPIC moving to the lower layers, while SENTI and NLI (and lesser degree QA), which require more complex intra-sentence interactions, move towards the higher layers. Recall that codelength and performance for SENTI, NIJI and QA also im- prove around the same time. These dynamics show that the 'traditional NLP pipeline' in LMs (Tenney et al., 2019) actually emerges quite late, and that tasks appear share layers for large duration of LM training before specializing later on. Given that LMs are frequently undertrained (Hoffmann et al., 2022), these continual shifts to task-specific sub- spaces highlight that probing single checkpoints from specific timesteps may miss important dy- namics regarding how tasks are represented with respect to each other.
with PoS and SEM, but less with DEP, potentially focusing on entity-level information, but less on the functional relationships between them. QA, NLI, and later SENTI, also share parts of their subspaces, while TOPIC is more distinct, having more overlap with the token-level NER and SEM tasks. While these patterns are already present at step 0, they become more pronounced during the early stages from 10k 100k steps, and then become weaker towards the end as the subspaces disentangle and specialize within the model. Overall, subspaces appear to be related in a linguistically intuitive way, sharing more information during the critical learn- ing phase. followed by later specialization.
These learning phases suggest that introducing linguistically motivated inductive biases into the model may most beneficial during the early crit- ical learning phase, rather than at the end, after which subspaces have specialized. This an im- portant consideration for multi- learning, where the prevalent approach is to train on related tasks after the underlying LM has already converged. As it is also often unclear which tasks would be ben- eficial to jointly train given target task, our subspace-based approach quantifying task simi- larity could provide a more empirically grounded relatedness measure than linguistic intuition.
5.3 Cross-task Interactions
The previous measures suggest that tasks could be grouped based on their shared learning dynam- ics. Our subspace-based approach for measuring cross-task SSAs (Figure 6) allows us to quantify this intuition. Comparing how much the subspaces of different tasks overlap provides more holistic picture of task-relatedness than relying on specific data instances, and further shows how these simi- larities change over the duration of training.
Practical Implications
Based on these learning dynamics, we next analyze their impact on downstream applications. As the previous results suggest, around 10k-100k steps already suffice to achieve the highest information gains for most tasks and reach close to 90% of final codelength and probing performance. At the same time, performance and subspaces continue chang- ing throughout training, even if to lesser degree. In order understand what is being learned in these later stages, we analyze finer-grained probe performance (Section 6.,,, whether these dynamics are domain-specine (Section 6.2), and what effects they have on full fine-tuning (Section 6.3).
Overall, angles between tasks follow linguistic intuitions: the syntax-related PoS and DEP sub- spaces span highly similar regions in the general representational space. COREF, at least initially, also exhibits high similarity to the syntactic tasks, likely exploiting the same features to resolve, e.g., pronominal matches. The NER subspace overlaps

---

601
107
14
(a) POS
(b) NER
(c) DEP
Figure 7: Class-wise Fl over LM Training Time for PoS, NER and DEP as measured on each task's 's dev split.
(standard deviation across seeds). For readability, classes are grouped according to Appendix A.2.
6.1 Class-wise Probing
Pig oltcoaa Sturzeent
First, take closer look at what is being learned during later LM training via class-wise Fl. For PoS (Figure 7a), most classes follow the general learning dynamics observed in Section 5, with no larger changes occurring beyond 10k steps. One outlier is the NOUN category, which continues to increase in performance while the other classes converge. Similarly for NER (Figure 7b), we ob- serve that most classes stagnate beyond 10k steps, but that events (EVENT), products (PROD), and es- pecially persons (PERSON) and organizations (ORG), see larger performance gains later on. What sets these classes apart from, e.g., determiners (DET) and pronouns (PRON), as well as times (TIME) and numbers (NUM), is that they represent open-domain knowledge. Performance improves as the LM observes more entities and learns represent them better, while the closed classes are acquired and stabilize early on.
80
F1 (macrr)
Po5
Dep
NER
40
Cortt
Scnd
104
10"
Steps
Figure 8: OOD F1 (macro) over LM Training Time with dark/light areas indicating 95%% of maximum performance (standard deviation across seeds).
split OntoNotes 5.0 (Pradhan et al., 2013) into docu- ments from nw (newswire), bn (broadcast news), mg (magazines) for in-domain training. and bc (broad- cast conversations), tc (telephone conversations), wd (web data) for OOD evaluation, based on the assumption that these sets are the most distinct from each other. In addition, we use English Twee- bank v2 (Liu et al., 2018) for DEP, and TweetEval (Rosenthal et al., 2017) for SENTI. The learning dynamics in Figure 8 show that transfer scores are generally lower, but that previously observed trends such as the steep learning phase between 1k and 10k steps, and NER's steeper incline compared PoS and DEP continue hold. SENTI sees an over- all greater Fl increase given more training than in the in-domain case. From this view, there are how- ever no obvious phases during which only OOD performance increases. Instead, the OOD tasks seem to benefit from the same types of information as the in-domain tasks.
these classes likely
Turning to DEP, we observe similar contin- ued improvements for, c.g., nominal relations, while functional has already converged. In addi- tion, these class-wise dynamics further reveal that clausal relationships converge later than other relations, towards 100k steps. At the same time, modifier relations also stronger gains starting 100k steps. Both coincide with the performance and codelength improvements of QA. NLI and es- pecially SENTI, as well as with the layer depth and subspace specializations. We hypothesize that this is another learning phase during which the existing lower-level syntactic knowledge is embedded in better long-range contextualization.
6.3 Full Fine-tuning
6.2 Domain Specificity
Finally, in terms of downstream applicability, we evaluate the effect of pre-training duration fully fine-tuned model performance. Due the higher computational cost, we conduct sweep over a subset of checkpoints for which the probing experi- ments indicated distinctive characteristics: Starting
Next, we investigate whether the previous findings hold across domains, and whether LM training du- ration has an effect on cross-domain transferability by gathering oft-of-domain (OOD) evaluation sets for five of the tasks. For POS, NER and COREF, we

---

100
100
Re5
Dep
(macrr)
Sem
(macro)
60
Topk
Cord
Send
Sent
DA
20
0.
10 10a
102
10
10"
307
10a
10a
10a
107
10
Steps
Steps
Figure 9: Fl (macro) after Full Fine-tuning of LMs initialized from checkpoints across LM pre-training time, as measured on each task's dev split (standard deviation across seeds too small to plot). Altered score range for readability.
Figure 10: OOD F1 (macro) after Full Fine-tuning of LMs initialized from checkpoints across LM pre- training time, as measured on each task's dev split (stan- dard deviation across seeds too small to plot).
from scratch at O steps, at k steps before the crit- ical learning phase begins, at 5k steps around the steepest incline, at 20k after most growth plateaus, and at the final 2M step checkpoint (training details in Appendix B.3). The results in Figure show generally higher performance than for probing, as is to be expected. Interestingly, for the majority of tasks, full fine-tuning follows the same learn- ing dynamics as probing suggests-exhibinne exhibiting the greatest improvements in the 1k-20k range.
For example, within the multilingual OSCAR cor- pus (Ortiz Suárez et al., 2020), at least 60 languages (~~0%% reach this threshold.
Conclusion
Our subspace-based approach to analyzing linguis- tic information across LM training has yielded deeper insights into their Iearning dynamics: In addition the critical learning phase from 1k-10k steps, indicated by probing (Section 5.1) and fully fine-tuned performance (Section 6.3), our informa- tion theoretic approach identifies how linguistic subspaces continue changing, even if performance suggests otherwise (Section 5.2). For interpretabil- ity studies using single-checkpoinl probing, this is crucial, the information identified from a model may not be representative of final subspaces, es- pecially the model undertrained (Hoffmann et al., 2022). Leveraging probes as characteriza- tions task-specific subspaces further allows us to quantify cross-task similarity, and surfaces how information is shared according to a linguistically intuitive hierarchy. This is particularly prominent during the critical learning phase, followed by later specialization, more open-domain knowledge acquired and the contextualization ability of the encoder improves. For multi-task learning, these dynamics imply that information sharing may be most effective early on, but more difficult after subspaces have specialized (Section 5.3). Finally, our analyses of OOD and full fine-tuning corrob- orate the previous learning dynamics (Section 6), showing that mid-resource languages could still benefit from pre-trained LM encoders a fraction of the full fine-tuning costs, while simultaneously highlighting which information gains (i.c., open- domain, reasoning) require larger amounts of data.
While performance for most tasks is within the 90-95% range at 20k steps, starting full fine-tuning from the point of steepest information gain at 5k steps does not seem to suffice to reach this tar- get. Pre-training beyond 10k steps therefore seems crucial in order for the LM encoder become beneficial. While it is possible to reach final per- formance on PoS starting from a random encoder, even the similarly syntactic DEP sees substantial improvements up to the last pre-training step, likely connected to the improvement in longer-range con- textualization observed in Section 6.1. Similar patterns can be observed for the other tasks, and especially for QA which only reaches usable per- formance beyond 20k steps. In terms of out-of- domain generalization for full fine-tuning, Figure shows that continued pre-training does increase OOD performance, especially for more complex tasks. As such, LM pre-training, in combination with full fine-tuning, appears to be beneficial start- ing at around 10k steps for most traditional NL. tasks, followed by slower, but continued increases thereafter. As 10k updates constitute only 0.5% of full training, 133M observed subword to- kens, mid-resource languages may still benefit from Language-speciffe pre-trained LM encoders, even without access to English-level amounts of data.

---

Limitations
Broader Impact
Despite aiming for high level of granularity, there are certain insights for which we lack com- pute and/or training statistics. Obtaining the high- est resolution would require storing instance-level, second-order gradients during LM training, in or- der to identify how each data point influences the model (similarly to Achille et al., 2019). Neither our study, nor other checkpoint releases contain this information, however we release intermedi- ate optimizer states from our own LM training to enable future studies the gradient-level.
As this study examines the learning dynamics of LM representational spaces, its findings have wider downstream implications. Here, we specifically focus on three: First, for model interpretability, we identified that single-checkpoint probing does not provide the full picture with respect to how task- specific representations may change over time (e.g.. layer depth in Section 5.2). This is critical when probing for sensitive information in LMs (e.g.. bias), as representations may shift substantially, decreasing the effectiveness of interventions such as null-space projection (Ravfogel et al., 2020).
Another consideration is the complexity of our probes. To enable cross-task comparisons, we de- liberately restricted our probes to linear models, as any non-linearities make it difficult to apply sub- space comparison measures. As such they may not be able capture more complex information, such as for QA and NLI. By using information theoretic probing, we are able to observe that task-relevant information is still being learned, albeit to lower degree than for the other tasks. To nonetheless measure codelength for these more complex tasks, the same variational framework could be applied to non-linear models (Voita and Titov, 2020), however the resulting subspaces would also be non-linear, negatively impacting comparability.
Second, we see a potential for multi-task learn- ing to benefit from better understanding of cross- task learning dynamics. Often it is unclear which task combinations boost/degrade each others' per- formance and what the underlying reasons are. Our findings suggest that similarities of task subspaces generally follow linguistic intuitions, but that there are distinct phases during which they share more or less information. Later specialization appears to be particularly important for more context-sensitive tasks, but may also make multi-task training more difficult, as tasks share less information this stage. A question for future work may therefore be whether carly-stage multi-task learning could lead to better downstream performance.
As MultiBERTs forms our underlying LM archi- tecture, more research is needed to verify whether the observed dynamics hold for larger, autoregres- sive LMs. While we believe that these larger mod- els likely follow similar learning dynamics, extract- ing comparable subspaces will be even more dif- ficult as scale increases. In initial experiments, we investigated checkpoints of the LLMs listed in Section 2, however similarly finding that perfor- mance had already converged at the earliest avail- able checkpoint. Furthermore, checkpoints lacked information regarding how much data, i.e., sub- word tokens, had been observed each step.
Third, for learning from limited data, this work identifies distinct phases during which different types of linguistic information are Iearned, as well as what their effects on fully fine-tuned and OOD performance are. We especially hope that the early acquisition of large amounts of information rele- vant for many traditional NLP tasks, encourages practitioners to revisit training encoders for under- resourced languages, despite the current trend to- wards larger models.
Acknowledgements
Finally, we would like to highlight that this study is correlatory, and that practitioners interested in a particular task should verify its specific learning dynamics by training LMs with targeted interven- tions (e.g., Lasri et al., 2022; Chen et al., 2023; Hanna et al., 2023). For this work, such interven- tions would have been out of scope, as we aim for wide task coverage to analyze interactions between them. Nonetheless, we hope that our findings on these learning dynamics can inform future task- specific studies of this type.
Thanks to the NLPnorth, MaiNLP and y Edinburgh labs for their insightful feed- back, particularly Joris Baan, Mike Zhang, Naomi Saphra, Verena Blaschke, Verna Dankers, Xinpeng Wang as well to ITU's High-performance Com- puting Team. Additional thanks to the anonymous reviewers for their comments. This work is sup- ported by the Independent Research Fund Denmark (DFF) Sapere Aude grant 9063-00077B and the ERC Consolidator Grant DIALECT 101043235.

---

References
Alexis Conneau, German Kruszewski. Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into single S&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 21 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics.
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hes- sel Haagsma, Rik van Noord, Pierre Ludmann, Duc- Duy Nguyen, and Johan Bos. 2017. The Parallel Meaning Bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pa- pers, pages 242-247, Valencia, Spain. Association for Computational Linguistics.
Marie-Cane Catherine de Marneffe. Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter. Joakim Nivre, and Christopher D. Manning. 2014. Universal Stan- ford dependencies: A cross- linguiiii typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (I.REC 14), pages 4585-4592, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Lasha Abzianidze and Johan Bos. 2017. Towards uni- versal semantic tagging. In /WCS 2017 12th Inter- national Conference on Computational Semantics
Short papers.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless C. Fowikes, Stefano Soatto, and Pietro Perona. 2019. Task2vee: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV).
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representa- tions.
Mário Figueiredo. 2001. Adaptive sparseness using jeffreys prior. In Advances Neural Information Processing Systems, volume 14, MIT Press.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Computing Research Reposi- tory, arw.v:2304.011..00
John Firth. 1957. A synopsis of linguistic theory, 1930- 1955. Studies in linguistic analysis, pages 10-32.
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium. Association for Computational Linguistics.
Version 1.
Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. pages 3531-3541, Osaka, Japan. The COL- ING 2016 Organizing Committee.
Jihun Hamm and Daniel D. Lec. 2008. Grassmann discriminant analysis: A unifying view on subspace- based learning. In Proceedings of the 25th Interna- tional Conference on Machine Learning, ICML '08, page 376-383, New York, NY, USA. Association for Computing Machinery.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015, A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Michael Hanna, Roberto Zamparelli, and David Mareček. 2023. The functiionl relevance of probed information: A case study. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 835- 848, Dubrovnik. Croatia. Association for Computa- tional Linguistics.
Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. 2023. Sud- den drops in the loss: Syntax acquisition, phase tran- sitions, and simplicity bias in mlms. Computing Research Repository, arxiv:2309.0731. Version 1.
Cheng- Han Chiang, Sung-Fens Huang, and Hung-yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6813 13-6888, On- line. Association for Computational Linguistics.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers. Pauli Virtanen, David Cour- napeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark

---

Wiebe, Pearu Peterson. Pierre Sérard-Marchann Kevin Sheppard. Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. Nature, 585(7825)335--36..
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331-339.
Karim Lasri, Tiaoo Pimentel, Alessandro Lenei, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8818-8831 18-8811, Dublin, Ireland. Association for Compu- tational Linguistics.
Zellig S Harris. 1954. Distributional structure. Word, 10(2-3): 146-162.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nut- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame. Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021, Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, Thomas Hennigan, Fric Noland, Katherine Millican, George van den Driess- che, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Sys- lems, volume 35, pages 30016-30030. Curran Asso- ciates, Inc.
Antti Honkela and Harri Valpola. 2004. Variational learning and bits-back coding; an information- theoretic view to bayesian learning. IEEE transac- tions on Neural Networks, (500)8800-800.
Yijia Liu, Yi Zhu, Wanxiang Che, Bing Qin, Nathan Schneider, and Noah A. Smith, 2018. Parsing tweets into Universal Dependencies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 965-975, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Ha- jishirzi, and Noah A. Smith. 2021. Probing across time: What does RoBERTa know and when? In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 820-842, Punta Cana, Dominican Republic. Association for Computational Linguistics.
J. D. Hunter. 2007. Matplotlib: A 2d graphics environ- ment. Computing in Science & Engineering, 9(3):90- 95.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Computing Re- search Repository, arxiv: 1412.6980. Version 9.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization, In International Confer- ence on Learning Representationss
Durk P Kingma, Tim Salimans, and Max Welling. 2015 Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc.
Christos Louizos, Karen Ullrich, and Max Welling. 2017. Bayesian compression for deep learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Andrew V Knyazev and Merico E Argentati. 2002. Principal angles between subspaces in a-based scalar product: algorithms and perturbation estimates. SIAM Journal on Scientific Computing. 23(6)) 2040.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics, 1333333
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Confer- ence on Learning Representations.
Mistral. 2022. Mistral - Large Scale Language Mod- eling Made Easy. https://nlp. http::///Ipsssssffrroem stanford. edu/ mistral/.

---

Ines Montani, Matthew Honnibal, Matthew Honni- bal, Sofie Van Landeghem, Adriane Boyd, Hen- ning Peters, Paul O'Leary McCann, jim geovedi, Jim O'Regan, Maxim Samsonov, György Orosz, Daniël de Kok, Marcus Blättermann, Duygu Alti- nok, Raphael Mitsch, Madeesh Kannan, Søren Lind Kristiansen, Edward, Raphaël Bournhoneqque, Lj Mi- randa, Peter Baumgartner, Richard Hudson, Explo- sion Bot. Roman, Leander Fiedler, Ryn Daniels, Wan- naphong Phatthiyaphaibum Grégory Howard, and Yohei Tamura. 2023. spaCy: v3.5.2.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-72568 Online. Association for Computational Linguisties.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural net- works with canonical correlation. Computing Re- search Repository, arxiv: (exiv:1806.05770 880.09559. Version 3.
Sara Rosenthal, Noura Farra, and Preslav Nakov, 2017. SemEval-2011 task 4: Sentiment analysis in Twitter. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-2011) pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguistics.
Pedro Javier Ortiz Suárez, Laurent Romary, and Benoit Sagot. 2020. A monolingual approach contextual- ized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 1703- 1714, Online. Association for Computational Linguis- tics.
Naomi Saphra and Adam Lopez. 2019, Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3257-3267, Minneapolis. Minnesota. Association for Computa- tional Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai. and Soumith Chintala. 2019. PyTorch: An imperative style, peghpperformanc deep learning library. In Advances in Neural Information Process- ing Systems 32. pages 8024-8035. Curran Associates, Inc.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisherg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong. Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Ed- vardo González Ponferruda, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli. Hady Elsahar, Hamza Benyamina, Hicu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joy- deep Bhattacharjee. Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey. Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Max- imin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nu- rulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas. Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest,
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation 12). pages 2089- 2096, Istanbul. Turkey. European Language Re- sources Association (ELRA).
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria, Association for Computational Linguistics.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Sveea: Singular vec- tor canonical correlation analysis for deep learning dynamics and interpretability. Computing Research Repository, arxiv: 106iv:1000..00 05806. Version 2.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.

---

Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei. Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, So- maieh Nikpoor, Stanislav Silberberg, Suhas Pai, Syd- ney Zink, Tiago Timponi Torrent, Timo Schick, Tris- tan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepereq. Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si. Davut Emre Taşar, Eliz- abeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chhablani, Han Wang. Harshit Pandey, Hen- drik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben- David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vīkas Raunak, Xiangru Tang. Zheng-Xin Yong. Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jae- sung Tae, Jason Phang, Ofrr Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang. Mohammad Shoeybi. Myriam Peyrounette. Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San- chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurêlie Névéol, Charles Lover- ing. Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina. Eli Bog- danov, Genta Indra Winata, Hailey Schoelkopf, Jan- Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na- joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang. Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenčk Kasner, Al- ice Rueda, Amanda Pestana, Amir Feizpour, Am- mar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Are- zoo Abdollahi, Aycha Tammour, Azadeh HajiHos- seini, Bahareh Behroozi. Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contrac- tor, David Lansky, Davis David, Douwe Kicla, Duong A. Nguyen, Edward Tan, Emi Baylor, Ez- inwanne Ozoani, Fatima Mirza, Frankline Onon- iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim EI- badri, Margot Mieskes, Marissa Gerehick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Ra-
jani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Al- izadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le. Tobi Oyebade, Tricu Le, Yoyo Yang, Zach Nguyen, Abhinay Ramesh Kashyap. Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-rscaladda, Ayush Singh, Benjamin Beilharz, Bo Wang. Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñinn Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang. John Giorgi, Jonas Golde, Jose David Posada, Karthik Ranga- sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pàmies. Maria A Castillo, Mari- anna Nezhurina, Mario Sänger, Matthias Samwald. Michael Cullan, Michael Weinberg, Michiel De Wolf. Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung. Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg. Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra. Sid Ki- blawi, Simon Ott, Since Sang aroonsiri, Srishti Ku- mar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Ya- nis Labrak, Yash Shailesh Bajaj, Yash Venkatraman. Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b-parameter open-access multilingual language model. Comput- ing Research Repository, Ttt::2211.05100 Version 3.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language mod- els mirage? Computing Research Repository, arxiv:2304.1500 Version 1.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Ture, Jacob Eisenstein, Dipanjan Das. and Ellie Pavlick. 2022. The multiB- ERTs: BERT reproductions for robustness analysis. In International Conference on Learning Representa- rions.
Natalia Silveira, Timothy Dozat, Marie-Catherin Catherine de Marneffe, Samuel Bowman. Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (IRECC-014))
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 1-31-1642, Seattle, Washington, USA. Association for Computational Linguistics.

---

Ryan Techan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. 2022. Emergent structures and training dynamics in large language models. In Pro- ceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Lan-
Appendix
A Data Setup
A.1 Language Modeling
BookCorpus (Zhu et al., 2015) was originally collected to train sentence embedding model for aligning passages in books to scenes in their movie adaptations. It reportedly contains around I Ilk books in 16 genres, both stemming from the pub- lic domain as well as more contemporary works. In total, the corpus contains 74M pre-tokenized sentences. In our experiments, we use the public version from the HuggingFace Dataset Hub (Lhoest et al., 2021) which is available as bookcorpus.
guage Models, pages 146-159, Dirlual+Dublin. As-
sociation for Computational Linguistics.
lan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguisties.
Julia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. Computing Research Repository, 1axiv:1908.0885 Version 2.
English Wikipedia (Wikimedia, 2022) was used for training both original BERT (Devlin et al., 2019) as well as MultiBERTs (Sellam et al., 2022), however neither the version used, nor the pre- processing steps have been reported. In our ex- periments, we make use of the 20220301. en split of the wik wikipedia a dataset from the HuggingFace Dataset Hub (Lhoest et al., 2021). is available in a format where Wikipedia-specific markup has been removed, and each instance corresponds to the full text of an article. We further split these 6.5M articles into 144M sentences using spaCy v (Montani et al., 2023) and its en_core_web pre-processing pipeline.
Elena Voita and Ivan Titov. 2020. cotommttionthhoreette probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 183-196, Online, Association for Computa- tional Linguistics.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Computing Research Repository, arxiv:2206.07633 Version 2.
Wikimedia. 2022. Wikimedia downloads.
Pre-training Corpus The final LM pre-training corpus consists of 109M sentence pairs from Book- Corpus and Wikipedia, which are shuffled to be consecutive or randomly combined 50% of the time. They contain a total of 5.7B subword tokens of which 15% (and 80 at most) are replaced with a special [MASK] or another random token from the vocabulary. In practice, this results in 801M masked tokens, or 14.07% of the training data.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- iez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
A.2 Probing
OntoNotes 5.0 (Pradhan et al., 2013) contains documents from six domains, which we split into an in-domain and out-of-domain set for our anal- ysis in Section 6.2. Each sentence is annotated with multiple layers of which we use: parts-of- speech (POS), named entities (NER), and corefer- ence (COREF). It is split into 1 115,812 train, 15,680 dev, and 12,217 test instances.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 JEEÉ International Con- ference on Computer Vision (ICCV), pages 19-27.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT praiirann approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1227, Huhhot, China. Chinese Informa- tion Processing Society of China.
PoS follows the Penn Treebank schema (Mar- cus et al., 1993) with 51 classes. The class-wise analysis in Section 6.1 uses a mapping from this

---

labeling scheme the Universal Part-of-Speech tagset (Petrov al., 2012).
20 Newsgroups (Lang, 1995) contains email threads from 20 mailing lists, grouped by their TOPIC. Our experiments use the bydate-version which sorted by date and removes duplicate en- tries and email headers containing the topic title. As the official data does not contain a dev split, we subdivide the training data, resulting in 9,051 train, 2,263 dev, and 7,532 test instance.
NER covers 18 entity types which are labeled using BIO schema. For grouping these entities, create the following custom mapping:
EVENT: EVENT;
LANG: LANGUAGE;
Stanford Sentiment Treebank (Socher et al., 2013) contains movie reviews along with their constituency parses and SENTI labels. We use the binarized SST-2 version with positive/negatiiv labels. The dataset consists of 67,349 train, 872 dev, and 1,821 test instances.
LOC: GPE, LOC, NORP;
NUM: CARDINAL, DATE, MONEY, ORDINAL, PERCENT, QUANTITY;
TveetEval (Rosenthal et al., 2017) consitutes the OOD setup for SENTI. It contains seven anno- tation layers on Twitter data. We use the general sentiment labels and binarize them to match SST-2. The dataset consists of 45,615 (unused) train, 2,000 dev, and 12,284 test instances.
ORG: ORG, FAC;
PERSON: PERSON;
PROD: PRODUCT; WORK_OF _ART, LAW;
Stanford Question Answering Dataset (Ra- jpurkar et al., 2016) contains user-generan user-generated questions for which answer passages can be found in the corresponding Wikipedia articles, i.e., extrac- tive question answering (QA). In our experiments, question forms the first input sequence to the model, followed by separator [SEP] token, and the relevant Wikipedia passage. Tokens within the answer passage are labeled as I, while everything else is O. The dataset consists of 87,599 train, and 10,570 dev instances.
TIME:
TIME.
COREF is built by extracting sentences with self- contained coreferences. The coreferring tokens are labeled as I, while all other tokens are labeled as O.
English Web Treebank (Silveira et al., 2014) contains syntactic dependencies (DEP) from 36 classes. To linearize the task, each token is labeled with the relation to its head word. For grouping the dependency relations, from the official Universal Dependencies taxonomy (de Marneffe al., 2014). The dataset consists of 12,543 train, 2,001 dev, and 2,077 test instances.
the nine categories
Stanford Natural Language Inference Dataset (Bowman et al., 2015) contains premise- hypothesis pairs for natural language inference (NLI). Given a sentence pair, separated by [SEP], the task to predict whether the relation of the two inputs is an entai entailment, contradiction, or neutral. The dataset consists of 550,152 train, 10,000 dev, and 10,000 test instances.
English Tweebank v2 (Liu al., 2018) consti- tutes the OOD setup for speciifcill Twitter data. It adheres to the same Universal Dependen- cies relation label set as EWT, and consists of 1,639 (unused) train, 710 dev, and 1,201 test instances.
B Experimental Setup
Parallel Meaning Bank (Abzianidze et al., 2017) contains three semantic annotation layers, of which we use the Universal Semantic Tags (SEM; Abzian- idze and Bos, 2017). They denote 69 cross-lingual, lexical semantic categories the token level. For grouping these tags, we combine the taxonomies of (Bjerva et al., 2016) and (Abzianidze et al., 2017) into 14 higher-level categories. The dataset consists 7,745 train, 1, 1.4 dev, and 1,053 test instances.
B.1 Language Modeling
Architecture The LM architecture used in our experiments is MultiBERts (Sellam et al., 2022). which follows BERT base base (Devlin et al., 2019), i.e., 12 layers with d 768. We use the check- points published on HuggingFace Hub (Wolf al., 2020) under google/mul tiberts-seed__o-e [ep__oo2000k/ For our own early checkpoints,

---

we start from step_ 0 of each respective seed and train on the same data in the same order. From this training run, we store checkpoints and optimizer states at steps 10, 100-1,000 in increments of 100, 1,000-20,000 in increments of 1, 1,000, and 40,000 for overlap comparisons, resulting in 29 additional models per initialization.
gold labels. In addition, the KL divergence between 0's posterior 8 and its sparsity inducing prior y are minimized according to Equation to ensure maximum compression of both the data and the probe itself. Following (Voita and Titov, 2020), we use the Adam optimizer (Kingma and Ba, 2014) with 10-3 learning rate, 31 = 0.9, 82 0.999 and O weight decay. Probes are trained with batch size of 64 for maximum of 30 epochs, and with early stopping on the development data if losses do not decrease.
Training The LM training procedure for our own early checkpoints follows Sellam et al. (2022) as closely as possible. We use the Adam optimizer (Loshchilov and Hutter, 2019) with a 10-4 learning rate, 31 = 0.9, 32 = 0.999 and a 10-2 weight decay, The learning rate is coupled to a polynomial sched- ule with a 10 step warm-up and consequent decay until the end. Each batch contains 256 sentence pairs with maximum length of 512 subword to- kens. The model is trained to fill in masked tokens (MLM) using a language modeling head, as well as to predict whether one sentence follows another (NSP) based on the [CLS] token and separate linear classification head.
Evaluation Following Saphra and Lopez (2019), we probe for task-specific information at the subword-level, meaning that for token-level tasks each token label is repeated across all of its con- stituent subwords, while for sequence-level tasks, the sequence label is repeated across all subwords. This corresponds to identifying task-specific infor- mation that is consistent across all contextualized embeddings within a sequence. To evaluate perfor- mance, we use macro-F1, as it is easier to interpret overall class wise performance in-spite of class im- balances. E.g., NER and QA have a high number of O labels which are classified correctly with above 95% F1. With micro-F1, performance would ap- pear unreasonably high, even if no named entities or answers would be identified.
B.2 Probing
Architecture Each probe receives embeddings {ho..... hi E Idd from all layers (including the non-contextualized layer O) as input, and summa- rizes them into a learned weighted average using a E R following (Tenney et al., 2019). This rep- resentation h' = E'= hr,, is then multiplied by linear transformation O IR to produce logits for the output classes. Following the variational MDL formulation of Voita and Titov (2020), each parameter w in 0 is drawn from a normal distri- bution w ~ N(2ju, 2²a²) with learned mean and variance o². both scaled by 2. There is one & per input dimension d, which is is also drawn from normal distribution its own learned mean μl= and variance a. During training this process is made differentiable using the reparametrization trick (Kingma et al., 2015). Each w and pair is coupled to a joint normal- Jeffreys prior 7ww, 2) x J (wo 2²), according to Figueiredo (2001) and Louizos et al. (2017). It induces sparsity in 0 by encouraging values of which are close to zero and have low variance. While the probe could theoretically be made more specifically use
B.3 Full Fine-tuning
Architecture For the full fine-tuning experi- ments in Section 6.3, we train all parameters the LM encoders from steps O, 1k, 5k, 20k and 2M. For token-level tasks, we add a linear layer on top of the final contextualized embedding layer, while for sequence-level tasks, a linear layer is fed each input sequence's 's [CLS] token. These linear classi- fication heads have the same dimensionality the linear probes, but are not sampled variationally.
(M,, oz) with
Training The fully fine-tuned models are trained using cross-entropy loss. Based on the recommen- dations in Devlin et al. (2019), we set the learning rate of the Adam optimizer (Kingma and Ba, 2014) to 3 x 10-5, and retain the other hyperparameters. Models are trained for maximum of 30 epochs, with early stopping on the development data when the loss stagnates.
complex (i.e., non-linear), linear model in order to enable geometric compar- isons between the resulting subspaces.
B.4 Implementation
Implementations use PyTorch v1.13 (Paszke et al., 2019) and NumPy v v1.24 (Harris et al., 2020). Visu- alizations use matplotib v3.6 (Hunter, 2007). Mod-
Training Both a and are jointly optimized by minimizing cross cross-entropy between predictions and

---

10H4
Subword Toleets
104
10"
10
10
102
104
Stops
Steps
Figure 11: Number of Subword Tokens Ob- served during LM training, following the trajec- tory of our re-created dataset, plus the estimated upper/lower bounds for prior work.
Figure 12: MLM and NSP Losses during the pre- training procedure described in Section 4.1, measured for each batch from step 0 to 40k. Checkpoints, training statistics and optimizer states released on publication.
90
For our LM training runs from step 0 40k, Fig- ure 12 shows how NSP and especially MLM losses start decreasing already after 100 pre-training steps. In general, losses on the actual LM pre-training tasks appears to decrease earlier by one order magnitude, before probing performance and code- length improve.
80
Nu5
70
Dep
60
Sem
50
Topic
30
Senti
10f
10
10
107
10
Across LM training, task subspaces extracted at the same timestep, but across different seeds, are close orthogonal each other, as in Figure 13. This contrasts checkpoints starting from the same initialization (see Section 5.2), where the rate of change differs substantially across training, but generally remains under 60%, even when trained on slightly different data (Section 4.1).
Steps
Figure 13: SSAs across Random Initializations for task-wise probes at each timestep (standard deviation across pairwise comparisons).
els were trained on NVIDIA A100 GPU with 40GBs of VRAM and AMD Epyc 7662 CPU. Probe training takes 10-60 minutes per check- point, dependent on the size of the dataset. Data pre-processing for language modeling (i.e., sen- tence splitting, tokenization, sampling and mask- ing) takes 160 hours for the full dataset. The LM training process itself takes around 50 hours for 40k steps. The random seeds used in our experiments follow MultiBETTs (Sellam et al., 2022) and are: 0, 1, 2, 3, 4. Further, the code for reproducing our experiments is available http://gtthhacommmaaaiuuuuupepee
C.2 Layer Weightings
In addition to the center of gravity (CoG) measure used in Section 5.2, the full layer weightings for each task and timestep are shown in Figure 14. They provide a more detailed picture for cases in which multiple layers are weighted similarly, e.g., for the first checkpoints DEP (Figure 14b, SEM (Figure 1 14c) and NER (Figure 14d). Both the ear- lier and later layers are weighted strongly, meaning that probes are making use of non-con non-contextual plus mixed representations at these early stages. In con- trast, SENTI and NLI almost exclusively rely on the last layer, while COREF and QA initially spread out across all layer depths. Later on in train- ing, these weightings also exhibit the specialization observed in Section 5.2, however weights are more spread out and do not collapse onto a single layer. This is most prominent with POS, but also DEP, SEM, NER and CORFF, which all make use of information from across wider range of depths.
Additional Results
C.1 Language Modeling
Figure 11 plots the number of subword tokens ob- served by the LM over the course of training. While this corresponds to the statistics of our re-created LM training corpus, the fact that the curve lies ex- actly between the feasible upper and lower-bounds given batch size and minimun/maximun subword tokens per sequence, we estimate that the original models also followed this trajectory.

---

(a) POS
DEP
(c) SEM
NER
COREF
TOPIC
(g) SENTI
QA
NLI
Figure 14: Layer Weightings a over LM Training Time for all tasks, corresponding to the center of gravity measure in Section 5.2. Darker/lighter fields correspond to more/less weight respectively.


Reviews:
[{"review_id": "mvVgTSnKn2", "reviewer": "Reviewer_n5w2", "summary": "", "questions": "Question A: is there any support for consistency and saliency in Line 165-167\n\nQuestion B: when is the critical learning phase? Figure 2+3 have different critical learning phases than Figure 4+5.\n\nQuestion C: could you clarify what do you mean by “a well-motivated, linguistically related set of tasks” (Line 484)? How do you expect researchers to make use of those tasks?\n\nQuestion D: according to Fig 7c, the phrase seems to be around 10^4 (10k) instead of 100k?\n\nQuestion E: Section 6.3 What is the Full Fine-tuning task?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To address your primary concern regarding the comparability of MultiBERTs and our own reproduced checkpoints: We would like to highlight that i) we carefully checked our early checkpoints to make sure they are comparable by looking at them in two ways (see next paragraphs) while avoiding costly full re-training, and ii) we leverage MultiBERTs as a unique resource.\n\nIn more detail, we originally identified the need to retrain early checkpoints due to the earliest available checkpoints of MultiBERTs as well as multiple larger models listed in Section 2 (ll. 86–90) exhibiting close to the final levels of performance (see also Figures 2, 3, 7, 8 and 9). To do so within our available compute budget, we followed the original work as closely as possible in terms of data, architecture, hyperparameters (see Section 4), and by reaching out to original authors of MultiBERTs for advice.\n\nThe resulting reproduced models were trained up to step 40k, beyond the intersection at 20k steps, in order to verify that performance trajectories overlap. This continued trajectory is visible in all F1 and code length related plots (Figures 2, 3, 7 8, 9). Continuing to train for another 10^6 steps with all five seeds would have required approximately 9000 TPU hours (Sellam et al., 2022), which we deemed unnecessarily wasteful. In this sense, MultiBERTs are a unique resource in that they provide many later checkpoints across multiple seeds, while being based on a base architecture for which we can reproduce earlier checkpoints.\n\nWe additionally examined the representational similarity of the original and reproduced models (ll.289–311). This also relates to the gaps you mention for Figures 4 and 5. The subspace angles between the original and reproduced models are around 60 degrees (ll. 292–294), however this difference is comparable to angles observed within the same training run (see steps 10^2 and 10^3 in Figure 4). Furthermore, models from the same training run, but initialized using different seeds, have angles over 80 degrees (ll. 304–307), meaning that our reproductions are substantially closer to the original models than the same models are across seeds. Given additional space, and based on your suggestions, modifying the plots to show both the overlapping performance trajectories, as well as the subspace angles across different random initializations could be used to highlight the aforementioned observations.\n\n**Questions**\n\n> Question A: is there any support for consistency and saliency in Line 165-167\n\nInformation which can be extracted easily (i.e., by a linear probe with high accuracy) can be seen as salient, and consistent representations for members of the same class lead to more efficiency in this process. Information theoretic probing and its measure of code length allow us to measure both properties in a more theoretically rigorous manner than using accuracy alone.\n\n> Question B: when is the critical learning phase? Figure 2+3 have different critical learning phases than Figure 4+5.\n\nWe place the \"critical learning phase\" to be around 10^3–10^5 range, depending on the type of linguistic information. You are correct that the highest rate of change differs depending on the measure used. Sections 5.1, 5.2 and 5.3 therefore attempt to highlight different aspects of each.\n\n> Question C: could you clarify what do you mean by \"a well-motivated, linguistically related set of tasks\" (Line 484)? How do you expect researchers to make use of those tasks?\n\nIn multi-task learning, it is frequently unclear which tasks boost or harm each other’s performance. Our experiments in Section 5.3 and Figure 6 show that task similarity in LM representations follow linguistic intuitions, and our additional findings regarding how this similarity changes over time (see also R3), can inform future work on training multi-task models from intermediate checkpoints or from scratch.\n\n> Question D: according to Fig 7c, the phrase seems to be around 10^4 (10k) instead of 100k?\n\nThe largest rate of change is indeed at 10^4. We refer to the convergence of performance at around 10^5 (ll. 527–529).\n\n> Question E: Section 6.3 What is the Full Fine-tuning task?\n\nSince our probing experiments use simple classification models, which are regularized using the minimum description length objective (Section 3.2), our experiments in Section 6.3 use a standard full LM fine-tuning procedure to measure more realistic downstream performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper extends the previous work by using information-theoretic probes and examining the representation of transformers during training. This finer-grain and more theoretically supported (linear algebra analysis is theoretically more justified than accuracy) analysis provides stronger evidence even for a similar conclusion.\n\nDespite gaps in Fig 4+5, it’s surprising to see that Figure 2+3 has a relatively smooth transition between reproduction and MultiBERT. This interesting evidence shows that the acquisition of linguistic property is similar at the same training stage.", "reasons_to_reject": "The validity of putting BERT reproduction and MultiBERT in the same plot is questionable, as there are some gaps in Fig 4+5. \n\nSection 6 seems to be a miscellaneous section that contains further analysis, and it seems hard to make connections to downstream and more practical tasks.", "questions_to_authors": "Question A: is there any support for consistency and saliency in Line 165-167\n\nQuestion B: when is the critical learning phase? Figure 2+3 have different critical learning phases than Figure 4+5.\n\nQuestion C: could you clarify what do you mean by “a well-motivated, linguistically related set of tasks” (Line 484)? How do you expect researchers to make use of those tasks?\n\nQuestion D: according to Fig 7c, the phrase seems to be around 10^4 (10k) instead of 100k?\n\nQuestion E: Section 6.3 What is the Full Fine-tuning task?", "paper_topic_and_main_contributions": "This paper extends the previous work by using information-theoretic probes and examining the representation of transformers (across layer) during training. Building upon the learned probes, the author further measures the subspace expressed by the probes’ parameters.The main contribution of this work is that it gives a stronger theoretical support for previous conclusions. Encoding a linguistic property well means that it must be represented in a consistent and salient manner; a more organized subspace of learned representation is easier to group their classes, in contrast to a subspace of random representation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 6 is confusing. There are too many small cells, so I can’t really see the dynamics of each cell. I wonder if it suffices to just have top-k most similar subspaces."}, "extra_text": {}}, {"review_id": "WQgus76omm", "reviewer": "Reviewer_USmm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In addition to proposing \"a new probing suite to extract task-specific subspaces\" to reveal \"some characteristics therein\", we would like to highlight that this work sheds light on prior findings from learning dynamics at a finer-grained level of detail, using a stronger theoretical foundation (Reviewer n5w2). Our method of comparing representational subspaces across tasks allows us to identify critical phases of LM pre-training during which linguistic information is shared between tasks (Reviewer 29hu), leading to very different representations than prior work has assumed (e.g., layer depth in Tenney et al., 2019).\n\n> The experiments just conducts on a single model, without involving different model types and sizes, for which experimental results are much more needed.\n\nRegarding your primary feedback: While our experiments are based on a single but unique architecture, namely MultiBERTs, it is important to note that despite current trends, BERT-style embeddings are still widely used, and that our study is furthermore conducted on not just one model, but across five initializations and a total of 2.6k probing experiments. We do agree that similar analyses would be highly valuable for additional model types and sizes (see Limitations, ll. 666–679). For the scope of this paper, we however first decided to evaluate the feasibility of our new subspace-based probing methodology in higher detail and within our computational budget. Our initial experiments using larger architectures (mentioned in Sections 2, 3 and 4.1) showed that re-training all of these models would have been required in order to understand early-stage training dynamics. Even just for MultiBERTs, this would have required 9000 TPU hours (Sellam et al., 2022), so we hope that future work training LMs will release more of their earlier checkpoints.\n\nWe would be glad to use an extra page to expand on the information theoretic methodology described in Sections 3.2 and 3.3. Since we do not wish to simply duplicate existing definitions from existing information theoretic probing work, any advice regarding how we may clarify specific parts would be greatly appreciated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work proposes a new probing suite to explore linguistic information during LM training and conducts some experiments to reveal some characteristics therein.", "reasons_to_reject": "1. The experiments just conducts on a single model, without involving different model types and sizes, for which experimental results are much more needed. \n2. The writing in this paper is not reader-friendly enough, e.g. section 3.2 and 3.3", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to explore how linguistic information in the representational space emerges, shifts and interacts during LM Training.\nThe main contribution is to propose a new probing suite to extract task-specific subspaces and compare them.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vf13hDxbHC", "reviewer": "Reviewer_29hu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "There are multiple contributions which underpin the findings of this paper, which we hope to clarify here: Saphra and Lopez (2019)’s work on LSTMs employs SVCCA, which requires representations to originate from the same input data. Cross-task analyses therefore require a single dataset to be annotated with multiple labels. This applies to all prior learning dynamics work, as the information learned for different individual properties are not directly comparable to each other.\n\nOur main contribution—and finding, as mentioned in your review—therefore lies in the ability to compare the linguistic information represented as subspaces of different tasks with different underlying datasets. This allows us to, for instance, identify the high amount of information sharing during the early stages of LM training (Section 5.3).\n\nWith respect to your other questions, there are some prior works studying the learning dynamics of transformer models (Chiang et al., 2020; Liu et al., 2021), however they do not conduct cross-task analyses due to the aforementioned limitations, and do not cover multiple seeds of an otherwise identical LM architecture. The latter point, while not the main focus of our work, is nonetheless interesting to analyze, as we find that despite the different initializations' similar F1/compression trajectories, they are representationally quite dissimilar to each other (ll. 304–307).\n\nFor OOD performance, we were also surprised to see that generalization ability does not necessarily increase during later stages. In separate experiments, we actually observed that this observation holds not just for our simple probing classifiers, but also for full fine-tuning.\n\nWe hope that these points clarify our main contributions and highlight their novelty. Our findings focus less on the tests for linguistic information, but rather on its emergence over the course of training, as well as how it relates to each other across tasks.\n\n**Clarifications**\n\n> the distributional hypothesis has \"scaled beyond semantics\" - what does that mean?\n\nWhile language modeling is based on distributional semantics, it has far outgrown its only use as a method for learning dense representations for downstream tasks and is now used for all kinds of tasks in LLMs. We do agree that this phrasing is confusing, and would therefore opt to remove this from the paper’s final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments appear to be well done.\n\nSee previous box - there are some interesting findings.", "reasons_to_reject": "I found the framing of the paper quite confusing - the authors don't make the novelty of the paper very clear. They describe earlier \"probing\" papers as identifying task-relevant information rather than \"actual linguistic information\", but in fact the papers that are cited as examples of the probing paradigm are ones where the goal is to decode linguistic categories such as the number of the subject (Giulianelli) or the depth of the three (Conneau), etc. This gets even more confusing when it turns out that many of the probes that the current papers use are \"tasks\" (such as NLI or QA) rather than \"actual linguistic information\" (such as POS tags).\n\nI think the niche that this paper is filling isn't introducing tests for linguistic information - there are dozens of studies analyzing linguistic properties (as opposed to task performance) - but rather analyzing how linguistic abilities emerge over the course of training. But even this is something that has been done before, as the related work section lays out - what is the particular contribution of this paper? Doesn't the Saphra and Lopez study already study the emergence of linguistic properties such as POS tags over the course of training? Is this about LSTMs vs Transformers? If so, is this really the first paper to look at a transformer's learning dynamics? Or perhaps the novelty is in the fact that the authors looked at 5 seeds instead of just one?\n\nThe paper reports on a large number of relatively superficial analyses - it would have been nice to dig a little deeper into some of these analyses, and leave out the ones that have more obvious outcomes (e.g. the fact that performance for all tasks improves as you train on more data).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the representations of BERT-like models at the early stages of learning. The framing of the paper focuses on the subspace similarity method - comparing classifiers that decode information from the model's internal representations - but that analysis in fact ends up being a relatively minor part of the results.\n\nThere are some interesting observations about the timecourse of the emergence of linguistic information over the course of training, though nothing spectacular - for the most part, the models performance on the tasks increases most sharply between 1k and 10k steps, followed by a more gradual increase between 10k and 100k. This generally makes sense, because most linguistic tasks have an easy \"head\" and a hard \"tail\", so you can get pretty good accuracy with a not-very-strong model, but it's hard to get very good accuracy. This may also explain the (interesting) fact that there is more similarity across task subspaces in the earlier stages of training - there are coarse features that can get you pretty far on a lot of tasks.\n\nThe fact that the probe's compression rate improves at the later stages of training, even when F1 stays constant, is quite interesting (and seems vaguely related to the phenomenon of \"grokking\"). I thought better compression would also translate into better OOD performance but it doesn't seem to, at least not in the specific datasets used in this paper.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I couldn't follow the statement in the introduction that the distributional hypothesis has \"scaled beyond semantics\" - what does that mean?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a suite of probes for understanding when and where linguistic information (syntax, semantics, reasoning) is learnt in the embedded representations of MultiBERT models, including an information theoretic analysis supporting the specific formulation of the probes. The main concern shared by all reviewers is that the paper was often confusing and hard to follow. Some reviewers felt that treatment of highly related work was insufficient, some felt that the description of technical details was hard to follow, and some reviewers felt that the overall motivation/story of the paper was not clearly aligned with the provided results and analysis. If the paper is accepted, it should be substantially edited for clarity based on the feedback from the reviewers. After much clarification by the authors in the rebuttal, reviewers seemed to understand more, and found the results in the paper to be sound. While reviewers agreed that the topic was interesting and relevant, reviewers were not unanimously excited, citing lack of breadth in terms of models analyzed, and to much breadth (and lacking depth) in terms of the analysis performed.
Decision: Accept-Findings