Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Authors: Angelica Chen
Ravid Shwartz-Ziv
Kyunghyun Cho
Matthew L Leavitt
Naomi Saphra
Venue: ICLR 2024 spotlight (2024)
Keywords: interpretability
BERT
syntax
phase changes
simplicity bias
training dynamics
Primary area: visualization or interpretation of learned representations

Abstract: Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model. However, certain insights into model behavior may only be accessible by observing the trajectory of the training process. We present a case study of syntax acquisition in masked language models (MLMs) that demonstrates how analyzing the evolution of interpretable artifacts throughout training deepens our understanding of emergent behavior. In particular, we study Syntactic Attention Structure (SAS), a naturally emerging property of MLMs wherein specific Transformer heads tend to focus on specific syntactic relations. We identify a brief window in pretraining when models abruptly acquire SAS, concurrent with a steep drop in loss. This breakthrough precipitates the subsequent acquisition of linguistic capabilities. We then examine the causal role of SAS by manipulating SAS during training, and demonstrate that SAS is necessary for the development of grammatical capabilities. We further find that SAS competes with other beneficial traits during training, and that briefly suppressing SAS improves model quality. These findings offer an interpretation of a real-world example of both simplicity bias and breakthrough training dynamics.

SUDDEN DROPS IN THE LOSS: SYNTAX ACQUISITION, PHASE TRANSITIONS, AND SIMPLICITY BIAS IN MLMS
Angelica Chen'
Ravid Shwartz-Ziv Kyunghyun Cho
Matthew L. Leavitt'
Naomi Saphra
fangelica chen, serrid.swwart.... ziv, kyunghyun, cho) @nyu. edu
matchew@datologyai.. com nsaphra@fas fas. harvard. edu
NYU GGeentecch $CIFAR LMB
DatologyA1
Kempner Institute, Harvard
ABSTRACT
Most interpretability research NLP focuses on understanding the behavior and features of fully trained model. However, certain insights into model behavior may only be accessible by observing the trajectory of the training process. We present case study of syntax acquisition in masked language models (MLMs) that demonstrates how analyzing the evolution of interpretable artifacts throughout training deepens our understanding of emergent behavior. In particular, we study Syntactic Attention Structure (SAS), a naturally emerging property of MLMs wherein specific Transformer heads tend focus on specific syntactic relations. We identify brief window in pretraining when models abruptly acquire SAS, concurrent with steep drop in loss. This breakthrough precipitates the subsequent acquisition of linguistic capabilities. We then examine the causal role of SAS by manipulating SAS during training, and demonstrate that SAS is necessary for the development of grammatical capabilities. We further find that SAS competes with other beneficial traits during training, and that briefly suppressing SAS improves model quality. These findings offer interpretation of a real-world example of both simplicity bias and breakthrough training dynamics.
INTRODUCTION
While language model training usually leads to smooth improvements in loss over time (Kaplan et al., 2020), not all knowledge emerges uniformly. Instead, language models acquire different capabilities different points in training. Some capabilities remain fixed (Press al., (023) while others decline (McKenzie et al., 2022), as a function of dataset size or model capacity. Certain capabilities even exhibit abrupt improvements-tthn paper focuses on such discontinuous dynamics, which are often called breakthroughs (Srivastava et al. 2022), emergence (Wei et al. (022). breaks (Caballero et al.) 2023). or phase transitions (Olsson et al.) 2022). The interpretability literature rarely illuminates how these capabilities emerge, in part because most analyses only examine the final trained model. Instead, we consider developmental analysis
complementary explanatory lens.
To better understand the role of interpretable artifacts in model development, we analyze and manipulate these artifacts during training. We focus on case study of Syntactic Attention Structure (SAS), model behavior thought to relate to grammatical structure. By measuring and controlling the emergence SAS, we deepen our understanding of the relationship between the internal structural traits and extrinsic capabilities of masked language models (MLMs).
SAS occurs when a model learns specialized attention heads that focus on word's syntactic neighbors. This behavior emerges naturally during conventional MLM pre-training (Clark et al. 2019 Voita ctt al. 2019: Manning et al.) 2020)). We observe an abrupt spike in SAS at a consistent point in training. and explore its impact MLM capabilities by manipulating SAS during training. Our observations paint a picture of how interpretability artifacts may represent simplicity biases that compete with other learning strategies during MLM training. In summary,
main contributions are:
Monitoring latent syntactic structure (defined in Section 2.I (Section 4.1) precipitous loss drop composed of multiple phase transitions (defined in Section 2.3
throughout training, we identify

---

relating to various linguistic abilities. At the onset of this stage (which call the structure onset), SAS spikes. After the spike, the model starts handling complex linguistic phenomena correctly, as signaled by a break in BLiMP score (which we call the capabilities onset). Although the functional complexity of the model declines for the rest of training, it increases between these breaks.
We introduce regularizer to examine the causal role of SAS (defined in Section 2.2) and use it to
show that SAS is necessary for handling complex linguistic phenomena (Section 4.2) and that SAS
competes with an alternative strategy that exhibits its own break in the loss curve, which we call the alternative strategy onset.
Section 4.3 shows that briefly suppressing SAS improves model quality and accelerates convergence. Suppressing past the alternative strategy onset damages performance and blocks SAS long-term, suggesting this phase transition terminates a critical learning period.
0.4
Mean BLMP Accuracy
8
0.3
0.7
Loss
UAS
0.2
0.6
4
0.1
0.5
0
5
10
Step
x10*
Step x10
Step
x10*
Figure 1: BERT first learns focus on syntactic neighbors with specialized attention heads, and then exhibits grammatical capabilities in its MLM objective. The former (internal) and the latter (external) model behaviors both emerge abruptly, at moments we respectively call the structure onset (A) ) and capabilities onset (quantified as described in Section 2.3p. We separately visualize three runs with different seeds, noting that these seeds differ in the stability of Unlabeled Attachment Score (UAS; Section 2.Ip after the structure onset, but uniformly show that SAS emerges almost entirely brief window of time. We show (a) MLM loss, with 95% confidence intervals across samples bynonparametric bootstrapping: (b) internal grammar structure, measured by UAS on the parse induced by the attention distributions; and (c) external grammar capabilities, measured by
average BLiMP accuracy with 95% % confidence intervals across tasks by nonparametric bootstrapping.
2 METHODS
2.1 SYNTACTIC ATTENTION STRUCTURE
One proposal for interpreting attention is treat some attention weights as syntactic connections (Maniing et al. 2020 Voita et al.) 2019 Clark et al. 2019). Our method is based on Clark et al. (2019).
who find that some specialized attention heads focus on the target word's dependency relations.
Dependency parses describe latent syntactic structure. Each word in sentence has a word that modifies, which is its parent in the syntax tree. Each dependency labeled-e., e.g.. an adjective modifies a noun through an amod relation in the Universal Dependencies annotation system (Nivre) al. 2017). In the example that follows, when an MLM predicts the word nests, is likely to rely heavily on its syntactic relations builds and ugly. One head may attend to adjectival modifiers like ugly while another attends to direct objects like builds.
pol
My
bird
builds
ugly
nests
We call this tendency to form heads that specialize in specific syntactic relations Syntactic Attention Structure (SAS). To measure SAS, follow Clark al. (2019) in using simple probe based off
2

---

the surface-leve attention patterns, detailed in Appendix A The probe provides implicit parse, with an accuracy measured by unlabeled attachment score (UAS).
2.2 CONTROLLING SAS
In addition to training models with BERT Buse parameters, also train models where SAS is promoted or suppressed. The model with SAS promoted throughout training is called BERTsAS++ while the model with SAS suppressed throughout training is called BERT
In order to adjust SAS for these models, we train a BERT Base model through methods that are largely conventional (Section (.T). with one difference. We add syntactic regularizer that manipulates the structure of the attention distributions using a syntacticity score (x,,, equal to the maximum attention weight between syntactically connected words i and j. We use this regularizer penalize or reward higher attention weights on token's 's syntactic neighbors by adding it to the MLM loss LMLM- We scale the regularizer by constant coefficient which may be negative to promote SAS or positive to suppress SAS. If we denote (z) as the set of all dependents of z, then the new loss is:
L(x)= (IL (r)
(o 2y) BGED(E)
Σ
Original
Syntactic regularization
2.3 IDENTIFYING BREAKTHROUGIS
This paper studies breakthroughs sudden changes in model behavior during a brief window of training.
We use the term breakthroughs interchangeably with phase transitions, breaks, and emergence, as has been done in past literature (Olsson et al. 2022 Srivastava et al., [002] Wei et al. 2022 Caballero al. 2023). What do we consider to a breakthrough, given metric in timesteps) from initialization d? We are looking for break point d' with the sharpest angle in the trajectory of f, as determined by the slope between d' and d' + A for some distance A. If we have no measurements at the required distance, we infer value for f based on the available checkpoints- e.g., e.g., if d measured in discrete timesteps, we calculate the angle of loss at 50K steps for A- 5K by imputing the loss from checkpoints at 45K and 55K steps to calculate slope.
some distance (c.g..
break (f, A) = arg max (t + A) - f(0] - (f()) f(t A]
(2)
In other words, break(/, A) is the point that maximizes the difference between the slope from f(1) to f(t A) and the slope from f(t A) to ft),, approximating the point of maximum acceleration.
3 MODELS AND DATA
3.1 ARCHITECTURE AND TRAINING
We pre-train BERT Base models using largely the same training set-up and dataset as /Sellm al. (0022). We use the uncased architecture with 12 layers of 768 dimensions each and train with the AdamW optimizer (Loshchilov & Hutter 2019) for 1M steps with learning rate of Ic-4, 10,000 warm-up steps and training batch size of 256 on single 4 x 100 NVIDIA A100 node. Our results only consider checkpoints that are recorded while pretraining remains numerically stable for all seeds, so we only analyze up to 300K steps.
Our training set-up departs from the original BERT set-up (Devlin et al., 2019) in that we use fixed sequence length of 512 throughout training, which was shared by [Sellam et al. (202)). We also use the same WordPiece-based tokenizer as Devlin et al. (2019) and mask tokens with 15% probability. Unless otherwise stated, all experiments implemented with the HuggingFace transformers (v4. 12.5) (Wolf et al. 2020), Huggingface datasets (v2.7.1) (Lhoest et al. 2021 and Pytorch (v1.11) PPaskke al. 2019) libraries.
Our pre-training datasets consist of BookCorpus (Zhu et al. 2015) and English Wikipedia (Foundation 2022). Since do not have access to the original BERT pre- rre-training dataset, we use recent Wikipedia dump from May 2020. For pre-training runs where syntactic regularization is applied, we dependency parse Wikipedia with spacy (Honnibal & Montani 2017) for silver standard labels.
3

---

Intrinic Dimonsen
200
Mean Score
0.71
0.65
50
0.54
Step
x
Step
xie
(a) Intrinsic dimension (TwoNN) complexity. Alter- native complexity metrics in Appendix L.2
(b) Average finetuned GLUE score. Individual tasks in Appendix
Figure 2:
Metrics during BERT Rase training averaged, with 95% confidence intervals, across three onsets marked.
seeds. Structure (A) and capabilities
3.2 FINETUNING AND PROBING
Our fine-tuning set-up for each GLUE task matches that of the original paper (Wang et al. 2018), with initial learning rate le-4, batch size of 32, and 3 total epochs.
Fine-tuning GLUE
Evaluating BLiMP BLiMP WWarstadt al. 2020a is benchmark of minimal pairs for evaluating knowledge of various English grammatical phenomena. We evaluate performance using the MLM scoring function from Salazar et al. (2020) to compute the bseudo-log-likelibee sentences in each minimal pair, and counting the MLM as correct when it assigns higher value to the acceptable sentence in the pair. Further implementation details in Appendix D
of the
Evaluating SAS dependency parsing We measure SAS by evaluating the model's implicit best- head attention parse (Eq. (3) Clark et 2019) on a random sample of 1000 documents from the Wall Street Journal portion of the Penn Treebank (Marcus et al. (999), with the syntax annotation labels converted into the Stanford Dependencies format by Stanford Dependencies converter (Schuster & Manning 2016). We evaluate parse quality using the Unlabeled Attachment Score (UAS) computed from the attention map, as described in Eq.
RESULTS
Often, interpretable artifacts are assumed to be essential to model performance. However, evidence for the importance of SAS exists only at the instance level on single trained model. We know that specialized heads can predict dependencies (Clark et al. 2019) and that pruning them damages performance more than pruning other heads (Voita et al., 2019). However, these results only weak evidence that SAS is essential for modeling grammar. Passive observation of a trained model may discover artifacts that occur side effect of training without any effect model capabilities. Causal methods that intervene on particular components at test time (Vig et al.) 2020; Meng et al.) 2023), meanwhile, may interact with the rest of the model in complex ways, spuriously implying component to be essential for performance when it could be removed if were not so entangled with other features. They also only address whether component is necessary at test time, and whether that component is necessary during learning. Both test-time approaches-pasee passsiv observations and causal interventions are limited.
We begin by confirming the assumption that SAS must be essential to performance. To motivate the case for skepticism of the role of SAS, we note lack of correlation between SAS metrics and model capabilities across random pretraining seeds (Appendix E). After first strengthening the evidence for SAS as meaningful phenomenon by taking model development into account, we then draw connections to the literature on phase transitions, simplicity bias, and model complexity.
4.1 THE SYNTAX ACQUISITION PHASE
Most work on scaling laws (Kaplan et al. 2020) presents test loss as a quantity that homogeneously responds the scale of training, declining by power law relative to the size of the corpus. In
4

---

the MLM setting, instead identify precipitous drop in the loss curve of BERT Base (Fig. consistently spanning 20K-30K timesteps of training across various random seeds. We now show how this rapid learning stage can be interpreted the composition of two distinct phase transitions.
((a)
The MLM loss drop occurs alongside the acquisition of grammatical capabilities in two consecutive stages, each distinguished by breaks defined by Eq. (2). The first stage aligns with the formation of SAS- -ee call this break in implicit parse UAS the structure onset. As seen in Fig. [ the
UAS spikes a consistent time during each run, in tandem with abrupt improvements in MLM loss (Fig. [(a) and finetuning metrics (Fig. 2())) Immediately following the spike, UAS plateaus, but the loss continues to drop precipitously before leveling off. The second part of this loss drop is associated with a break in the observed grammatical capabilities of the model, as measured by accuracy on BLiMP (Fig. [(c).. We call the BLiMP break the capabilities onset. We show similar trajectories on the MultiERT's (Sellam et al., 2022) reproductions (Appendix F)
By observing these phase transitions, we can see that the internal representation of grammar, in the form of syntactic attention, precipitates the external observation of grammatical behavior, in the form of correct language modeling judgements on linguistically challenging examples. This is not only a single breakthrough during training, but a sequence of breakthroughs that appear to be dependent on each other. We might compare this to the "checkmate in one" " BIG-Bench task, a known breakthrough behavior in autoregressive language models (Srivastava et al. 2022). Only at large scale can models accurately identify checkmate moves, but further exploration revealed that the model was progressing in linear fashion at offering consistently valid chess moves before that point. The authors posited that the checkmate capability was dependent on the ability to make valid chess moves, and likewise it seems we have found that grammatical capabilities are dependent on a latent representation of syntactic structure in the form of SAS.
We find that the existence of these phase transitions holds even when using continuous metrics (Appendix D]. in contrast to Schaeffer et al. (2023), who found that many abrupt improvements in capabilities are due to the choice of thresholded metries like accuracy. We also find that the phase transitions hold even when setting the x-axis to some continuous alternative to discrete training timesteps, such as weight norm (Appendix(H) (H). Thus both x-axis and y-axis may use non-thresholded scales, and the phase transitions remain present.
Complexity and Compression The capabilities of language models have often been explained as a
form of compression (Chiang) 2023 Cho 2023 Sutskever 2023), implying continual decrease in functional complexity throughout training. A more nuanced view of training is suggested by work on critical learning periods (Achille et al. 2018) and the information bottleneck (IB) theory (Shwartz-Ziv & Tishby 20177), which tie phase transitions to shifts in complexity (Arnold et al. 2023). When we plot various complexity metrics (see Fig. 2(a) for TwoNN intrinsic dimension (Facco et al. 2017), and Appendix L.2 for additional metrics), we see a decline in complexity for most of training. However, an abrupt memorization phase of increasing complexity occurs between the structure and capabilities onsets, during which the model rapidly acquires information. (Although the mid-training timing of this memorization phase is novel, these dual phases are supported by the literature on IB and critical learning periods, as further discussed in Appendix GG) Taken together, see that the structure and capabilities onsets are not just phase transitions of internal structure and external capabilities, but also phase transitions of functional complexity.
The steep decrease in complexity that immediately precedes the structure onset (Fig. ((a)) supports the intuitive understanding that SAS must be simplistic in order to be human interpretable. Generally, models tend to favor simpler functions like SAS earlier in training (Hermann & Lampinen] 2020
al. 2019 Arpit et al. 2017). a tendency often referred to as simplicity bias. In Section 4.3 we use the simplicity bias literature to motivate our
Shah et al. 2020 Nakkiran et al. 2019 Valle-Péree
hypotheses about and interventions for competing strategies.
4.2 CONTROLLING SAS
Having established the natural emergence of SAS, we use our syntacticity regularizer (Section 2.2 to evaluate whether SAS is truly necessary for handling complex grammatical phenomena. We confirm that this regularizer can suppress or accelerate the SAS phase enhancing SAS behavior throughout training (BERTsAS+) leads to early improvements in ML
As seen in Fig.
(aa
5

---

10
Mean BLMP Accuracy
0.7
0.3
Loss
UAS
0.2
0.1
0.5
Step
x10
Step
x10
Step x10*
BERT
BERT sxs
BERT
Figure 3: Metrics over the course of training for baseline and SAS-regularize models (under both suppression and promotion of SAS). Structure (A) and capabilities onsets marked, except on BERT which does not clearly exhibit either onset. Each line is averaged over three random seeds. On y-axis: (a) MLM loss (b) Implicit parse accuracy (c) average BLiMP accuracy over all phenomena categories. Shaded regions represent the 95% confidence interval.
performance, but hurts later model quality Conversely, suppressing SAS (BERTsAs/ damages both early performance and long term performance. Suppressing SAS during training prevents the emergence of linguistically complex capabilities (Fig. In other words, preventing the internal grammar structure onset will also prevent the external grammar capabilities onset that follows it.
However, there exists an early apparent phase transition in the MLM loss (at around 6K steps), which suggests that an alternative strategy emerges that leads to improvements prior the structure onset. We therefore refer to this early inflection as the alternative strategy onset. Our results suggest that SAS is crucial for effectively representing grammar, but the existence of the alternative strategy onset implies that SAS also competes with other useful traits in the network. We explore the alternative strategy onset represented by the phase transition under SAS suppression Appendix M
Importantly, the break in the loss curve occurs earlier in training when suppressing SAS. The implication is profound: that the alternative strategy is competing with SAS, and suppressing SAS permits the model to learn the alternative strategy more effectively and earlier. Inspired by this insight, we next ask whether there can be larger advantages to avoiding the natural SAS-basee strategy early in training, thus claiming the benefits of the alternative strategy.
4.3 EARLY-STAGE SAS REGULARIZATION
Because BERTSAS, briefly outperforms both BERT Base and BERTSAS.. we have argued that suppress- ing SAS implicitly promotes competing strategy. This notion of competition between features or strategies is well- dell-docu in the literature simplicity bias (Shah al. 20200 Arpit et al.) 2017) Hermann & Lampinen] 2020; Pezeshki et al. 2021). Achille al. (2018) find that some patterns must be acquired early in training in order to be learned all, so depending too much on an overly simplistic strategy early in training can have significant long-term consequences on performance. To test the hypothesis that learning SAS early allows SAS to out-compete other beneficial strategies, this section presents experiments that only suppress the early acquisition of SAS. For multistage regularized models, we first suppress SAS with À = 0.001 and then set À = 0 after a pre-specified timestep in training. These models named after the timestep that SAS is suppressed until, e.g.,
( 5AS- is the model where is set to timestep 3000.
We find that suppressing SAS early improves the effectiveness of training later. Specifically, (3R)(3A)
SAS. outperforms BERT Base even well after both models pass their structure and capabilities onsets (Fig. 4(a) Table I although these advantages cease to be significant after longer training runs
Note that in BERT we see the capabilities onset is after the structure onset, but before the SAS plateau, suggesting that SAS only needs to hit some threshold to precipitate the capabilities onset, and does not need to stabilize.

---

2.1
0.45
UAS at steps
100K
2.9
0.33
2.8
0:22
1.7
0.10
no. steps suppressed
=10
steps suppressed
xio
0.75
0.80
100KK stops
GLUE at 100K
0.73
0.70
0.66
Max: no. steps suppressed
$10
Max. no. sceps suppressed
x10f
Figure 4: Metries for the checkpoint at 100k steps, for various models with SAS suppressed early in training. The vertical line marks the BERT alternative strategy onset; note that model quality is worst when the regularizer is changed during this phase transition. The x-axis reflects the timestep when the regularizer À is changed from 0.001 to O. To control for the length of training time without suppressing SAS, Appendix [ presents the same findings measured at a checkpoint exactly 50K timesteps after releasing the regularizer. On y-axis: (a) MLM loss; (b) Implicit parse accuracy
(UAS): (c) GLUE average (Task breakdown in Appendix ! (d) BL.MP average (Task break down in
Appendix K). Shaded regions represent 95% confidence intervals across three seeds.
MLM Loss 4
GLUE average 1
BLiMP average f
BERT
1.77 + 0.01
0.74 + 0.01
0.74 + 0.02
BERT SAS+
2.39 = 0.03
0.59 + 0.01
0.74 + 0.01
BERT SAS-
2.02 = 0.01
0.69 = 0.02
0.67 + 0.03
BERT (31)
1.75 + 0.01
0.74 0.00
0.75 + 0.01
Table 1: Evaluation metrics, with standard error, after training for 100K steps (~ 13M tokens), SAS as the best multistage hyperparameter setting based on MLM test loss at 100K steps. Bolded values significantly outperform non-bolded values in the same column under a 1-sided Welch's t-test.
averaged across three random seeds for each regularizer setting. We selected BERT!
(Appendix[P PP. Some multistage models even have more consistent SAS than BERT Base (Fig. 4(b).. posit that certain associative patterns are learned more quickly while suppressing SAS, and these patterns not only support overall performance but even provide improved features acquire SAS.
4.3.1 WHEN CAN RECOVER THE SAS PHASE TRANSITION?
Inspecting the learning curves of the temporarily suppressed models, we find that briefly suppressing SAS can promote performance (Appendix Nj and accelerate the structure onset (Fig. 5(a) while augmenting it (Fig, (())). However, after more prolonged suppression of SAS, it becomes impossible to hit the dramatic spike in implicit parse UAS that we in BERT (Section 4.3) If the SAS phase transition is prevented, MLM performance falls significantly compared to BERT Base and we SAS spike (Appendix N It appears that must choose between phase transitions: the model cannot undergo first the alternative strategy onset and then the structure onset. fact, we measure the worst model quality when switch settings during the alternative strategy transition (Fig. 4
7

---

X10
UAS saike at treak
0.3
Step of break
20.1
steps suppressed
xio
no. steps suppressed
x10*
(a) Timing of structure onset for models with tem- porary SAS suppression.
(b) Magnitude of UAS spike at structure onset time UAS,-
6, determined by UAS:+5000
Figure 5: SAS suppressed only briefly, accelerates and augments the SAS onset. However, further suppression delays and attenuates the spike in UAS, until it eventually ceases show clear inflection. A vertical dotted line marks the BERT SAS. alternative strategy onset and the shaded region indicates the 95% confidence interval across three seeds.
5 DISCUSSION AND CONCLUSIONS
Our work is a response to the limitations of probes that analyze only a single model checkpoint without regard its training history (Saphra 2023). We posit that developmental explanations, which incorporate model's training history, provide critical perspective and explanatory power. We have used this developmental approach to demonstrate the necessity of SAS for grammatical reasoning in MLMs, and have furthermore used SAS as case study to shed light on alleviating simplicity bias, the dynamics of model complexity, and the risks of changing the optimizer during phase transitions. Our work also guides further understanding of many deep learning phenomena, and may inspire more rigorous approach to science of deep learning. For further discussion Appendix presentss an extended review of related work causal interpretabilityy simplicity bias, and phase transitions.
5.1 EARLY DYNAMICS AND SIMPLICITY BIAS
Sutton 2019) introduced the machine learning world the Bitter Lesson: models that use informed priors based on domain understanding will always lose to generic models trained on large quantities of data. Our work suggests that we might go further: even generically learned structure a disadvantageously strong prior, that structure reflects human expert models of syntax. In other words, human interpretations of natural phenomena are simplistic that their presence early in training can serve as a negative signal. If this observation holds in natural language a modality that has evolved specifically be human interpretable how much worse might simplicity bias impact performance on other domains like scientific and physical modeling?
form
Dependency and competition We have found evidence of multiple possible relationships between emergent behaviors. Previous work suggests that model properties can depend on one another, e.g.. checkmate-ig in-one capabilities depend on first learning valid chess moves (Srivastava et al., [2022): ; or compete with one another, c.g., as sparse and dense representation strategies compete on arithmetic tasks (Merril et al. 2023). Similarly, we first find evidence of dependency relationship, based on our evidence that SAS is prerequisite for many linguistic capabilities indicated by BLiMP. Then, we identify a competitive relationship, based on our observations that suppressing SAS leads to an alternative strategy that prioritizes context differently. These distinct relationships shed light on how model behaviors interact during training and may suggest training improvements that delay or promote particular behaviors. Existing work in simplicity bias (Shah et al. 2020 Pezeshki et al.) 202I) suggests that preference for simple heuristics might prevent the model from acquiring more reliable strategy. Our results appear to be evidence of this pitfall in practice.
The benefits of early training without permitting SAS bear an intriguing parallel to pretraining. Just as pretraining removes the particulars of the downstream task by training generic language structure, early SAS suppression removes the particulars of linguistic structure itself. In doing encourage the MLM to treat the entire sequence without regard for proximity to the target word, as a ofggoof-ordd model might. Therefore, the beginning of training is even more unstructured and generic than it would be under the baseline MLM objective.
Pretraining
8

---

Curriculum learning We also offer some insights into why curriculum learning is rarely effective at large scales. Simple data is likely to encourage simplistic strategies, so any curriculum that homogenizes the early distribution could promote a simplistic strategy, helping early performance but harming later performance. Predictably, curricula no longer help at large scales (Wu et al. (021).
5.2 PHASE TRANSITIONS
Instability critical points Abrupt changes are rarely documented directly the level of validation loss, but we show that they may be observed- and interpreted in realistic settings. Smooth improvements in loss may even elide abrupt breakthroughs in specifie capabilities, as discussed in Appendix [..]] Our multistage results point surprising effect: that the worst time to change the regularization is during a phase transition. When we release the SAS suppression well before the point at which the alternative transition starts during BERT training (i.e., the alternative strategy onset), we find is possible to recover the SAS transition, preventing damage to GLUE, BLiMP, and MLM loss metrics. Likewise, although releasing the regularization after the alternative transition prevents the recovery of SAS, it nonetheless incurs limited damage to model quality metrics. However, releasing the regularizer during the phase transition leads a substantially worse model under every metric. These findings suggest that the moment of breakthrough constitutes critical point where an optimizer misstep can damage the performance of the model, possibly even at convergence. This phenomenon may be consequential for future optimization research.
5.3 INTERPRETABILIT EPISTEMOLOGY
While SAS was already known to emerge naturally in MLMs, there were reasons to be skeptical of its necessity. One objection is that raw attention distribution information is not guaranteed proxy for information flow (Abnar & Zuidema 2020; Ethayarajh & Jurafsky 202T). Another thread questions the interpretabilityy of attention by obfuscating the attention weights without damaging model performance (Jain & Wallace 2019 Serrano & Smith 2019). If the fundamentally informative nature of attention subject to extensive debate (Bibal et al. 2022), we must also be skeptical of overstating its connection to syntax. Attention syntacticity is microcosm of wider failures in the science of deep learning, which has been criticized for tendency to use aneedotal observations and post-hoc explanations, rather than rigorous correlational or causal tests (Forde & Paganini] 2019).
Prior evidence for the importance of SAS came in two forms, both of which operate post-hoc at the instance level on specific samples: instance-level observation in fully trained networks (Clark)
al., 2019) and instance-level causal experiments in fully trained networks (Voita et al., (019). Observational studies might discover structures that emerge side effect of training, rather than those crucial to the operation the model. Traits that emerge as a side effect of a process but appear crucial to performance
called spandrels in evolutionary biology; possible examples include human chins (Yong] 2016) and enjoyment of music (Pinker 1997). While instance- like Voita et al. (2019) may be epistemically stronger than the observational studies, the network's s failure to recover from a causal intervention does not indicate that it relies on the structure provided. Instead, the network may be more brittle large distribution shifts on the relevant features, without truly relying on those features (Tucker et al. 2021). One possible scenario is that a behavior may develop early in training and become vestigial (like a human's 's tailbone (Mukhopadhyay et al. 2012) but sufficiently integrated into subnetworks that generate and cancel information that the network cannot easily recover from its removal. To support the skeptical case, we find that SAS metries were not correlated with ML. capabilities across random seed (Fig.
causal experiments
We provide several epistemically strong results in favor of the importance of SAS. First, we study models in development (Section 4.1. finding that the SAS phase transition directly precipitates the emergence of linguistic capabilities. This result supports that blackbox grammatical capabilities depend measurable internal structures. Second, we have causal interventions on development (Section 4.2), which again reveal the importance of this head specialization behavior by promoting and suppressing it. Instance-level interpretability methods, at best, offer evidence that trait emerges and the model cannot recover from its removal; we can now say that certain capabilities depend on this trait although the model eventually discovers alternative ways to represent some of them.

---

ACKNOWLEDGMENTS
We thank Samuel Bowman and Jason Phang for helpful discussions during the development of this project.
This work was supported by National Science Foundation Award 1922658, Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling), and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to Al). Matthew Leavitt was employed at Mosaic ML and Naomi Saphra was employed NYU for the majority the work on this project.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4197, Online, July 2020. Association for Computational Linguisties. doi: 10. 10.188655/112222..ccmmm... 385. URL.
n.385
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrin- sic dimension of data representations in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.. fi less/pppe//219// http:////ppreeeeeennnneeere....p https/////ppppp....ee///////////
2019. URL b494449999999 e waddc334443be555cc Fi file/cfccee 111221149cc0
Julian Amold, Niels Lörch, Flemming Holtorf, and Frank Schäfer. Machine learning phase transitions: Connections to the fisher information, 2023.
Devansh Arpit, Stanislaw Jastrzębski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. Closer Look Memorization Deep Networks. arXiv: 05394 [cs, stat), June 2017. URL arXiv: 1706.05394.
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden Progress Deep Learning SGD Learns Parities Near the Computational Limit, July 2022.
URL httpp:://axx...og//aba..../
7082077.87999 MtXiv:2207.08799 [cs, math, stat].
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. arXiv preprint arXiv:2305. 156144 2023.
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang. Thomas François, and Patrick Watrin. Attention Explanation? An Introduction to the Debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3889-3900,, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/ v1/2022.acl-long 269. URL https: htt.:tppp//elaaatoooooooor/2222.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, 2023. URL https://
openreview.net. wtw.w.uneee.eooreeeee
forum? iidiiddsss wiw.mmiiisscccccc.cccc
David C. Chiang, Sung-Feng Huang, and Hung-yi Lee. Pretrained Language Model Embryology: The Birth of ALBERT. arXiv:20/0.0244 [cs], October 2020. URL [Rlp:tttp//////oorggg/
abs/2010.022 /2t./2200.022000
arXiv: 2010.02480.
Ted Chiang. ChatGPT Is Blumy JPEG of the Web. The New Yorker, February 2023, ISSN cww/tecc///cccccccccccccccccccct wwwehh/nnaalssae wot-offfttchhooooyy..oo -ea---s----llul ohatt
0028-792X. URL http:://w..wwwtt/tttttttoooo
Section: annals of artificial intelligence.
10

---

Kyunghyun Cho. Are JPEG and LM similar to each other? If in what sense, and is this the real ques-
tion to ask? http:///ss//aaaaaaaaaa/////////a
December 2023.
Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The Grammar- Leamnig
Trajectories of Neural Language Models. arXiv:2109.0600 060099 [cs], March 2022. URL http:
7/a //axxiv.org/bb///211 /2tp//009999996
arXiv: 2109.06096.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher Manning. What Does BERT
Look At? An Analysis of BERT's Attention, June 2019. URL http::/arxxvv.og///ba 1906.04341 arXiv:1906.04533 [cs].
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4154-4175, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1.11653/1//2222.ad BtRp:ttt:://eot /ttp/ttss//aaaaataoooooo////////
Jacob Devlin, Ming-Woil Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.11653/11/N19-14422 URL https 7/acl aatho ogy.org/N19-142.
Kawin Ethayarajh and Dan Jurafsky. Attention flows are shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th Interna- tional Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 49-54, On- line, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 acl-short.8. 8. URL
short.8
Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic di- mension of datasets by a minimal neighborhood information. Scientific Reports, 7(1), sep 2017. doi: 10. 10.1088/41198-007-1118888
Jessica Zosa Forde and Michela Paganini. The scientific method in the science of machine learning. CORR, abs/1904. 10922, 2019. (ttp://tp.////oloooo/00000013133
11873-y
Wikimedia Foundation. Wikimedia downloads, 2022. URL WtRL_tttss///umpswwkkkkee
Katherine Hermann and Andrew K. Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. arXiv:2006. 12433 [cs, stat], October 2020. URL http:
(laxxiv.org/abs/2000
arXiv: 1.006.1433..
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom
embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Michael Y. Hu, Angelica Chen, Naomi Saphra, and Kyunghyun Cho. Latent state models of training dynamics, 2023.
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. 1902 [cs], February 2019. httpttpp: httppp///aaavi..aaaba/////////// 902.10 0186,
arXiv: 1002.10186.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. arXiv:2002.099: 09572 [cs, stat), February 2020. URL http:ht:p//arrii.r////a///////// arXiv: 2002.09572.
Keller Jordan. Calibrated chaos: Variance between runs of neural network training is harmless and inevitable, 2023.
11

---

Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models.
arXiv:2001.088 0 [cs, stat], January 2020. URL arXiv: 2001.08361.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? arXiv preprini arXiv:2305. 18887, 2023.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. ArXiv, abs/1905.0044 2019. URL httpp://
Matthew L. Leavitt and Ari S. Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in dnns. ArXiv, abs/2003.01262. 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv:2003. 2003::22000002188 [cs, stat], March 2020.
arviv: 2003.02218.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Mai Major, Philipp Schmid. Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac. Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https demo..22
Zachary C. Lipton. The Mythos of Model Interpretability; In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31 June 2018. ISSN 1542-7730, 1542-7749. doi: 10. (45/3336388.32411333333 URL httpatpss//.......orrrro///////.
3236 6236386.3241340
Zeyu Liu, Yizhong Wang. Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does RoBERTa know and when?
Franni-rannenne Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 820-842, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2011 htt.://////22....nnnnggs... findings--nninn -mnpp..-.mmlp.1.... URL
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representations, 2019, /trll/ttpps////eeeeveeeeeeeeee/e
Bkg6RiCqY)
Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. Eval- uation beyond Task Performance: Analyzing Concepts in AlphaZero in Hex. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35. pp. 25992-26000 Curran Associates, Inc., 2022. URL hitps:tpppo.o. /tt.o///oeaaee//eeeeeeeeoooooeor http://ppeee.illeeepppppre////// ff222007777TB3301331000444422200
12

---

Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-suuprvisiio. Proceedings of the National Academy of Sciences, 117(48):00066330054 2020.
Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. Treebank-3, 1999.
URL http://ppw./aaaaao.......peeeeea
Thomas McGrath, Andrei Kapishnikov, Nenad Tomaš ev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in AlphaZero. Proceedings of the National Academy of Sciences, 119(47), nov 2022. doi:
10. (0773/pnas.2006225111 URL httt:tt:t///oooo http://..orr.....033322222222232
Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez. The inverse scaling prize, 2022a. URL hilps://github. com/inverse- scaling/prize, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks, 2023.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation, 2018.
Biswanath Mukhopadhyay, Ram M Shukla, Madhumita Mukhopadhyay, Kartik C Mandal, Pankaj Haldar, and Abhijit Benare. Spectrum of human tails: A report of six cases. Journal of Indian Association of Pediatric Surgeons, 17(1):23, 2012.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Characterizing Intrinsic Compositionality in Transformers with Tree Projections, November 2022. URL http:
larxiv.org/bb// http:///a//222211111111 22212022221181 ar arXiv:2211.0122 [cs].
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Grokking of hierarchical structure in vanilla transformers, 2023.
Prectum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. SGD on Neural Networks Learns Functions of Increasing Complexity. arXiv: /905. 1604 [cs, stat], May 2019. Utthttp::/// http:h::://aai.../////aaaa 1905.11604.
905.1 160411604
arXiv:
Neel Nanda and Tom Lieberum. A Mechanistic Interpretability Analysis of Grokking, 2022. http://tttt///oooaatoeettttttttt htt.:///ww.opooooooooooooo////io 7RQMKDhYjB/a bec.aaacce.....teppaaaaaaayayaa.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability, 2023, URL http:/ts////xxixo////////
23C1.05217
Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. Universal Dependencies. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Tutorial Abstracts, Valencia, Spain, April 2017. Association for Computational Linguistics. URL
attps::/aclammhhooooo..orr/
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, T. J. Henighan, Benjamin Mann, Amanda Askell, Yushi Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom B. Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Christopher Olah. In-context learning and induction heads. ArXiv, abs/2209. 11895. 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch An Imperative Style. High- Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, 2019.
13

---

Mohammad Pezeshki, Oumaar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guil- laume Lajoie. Gradient Starvation: A Learning Proclivity in Neural Networks. arXiv:2011.00 .94688
[cs, math, stat], November 2021. 2011.09468.
arXiv:
Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. Pareto Probing: Trading Off Accuracy for Complexity. In EMNLP, October 2020. URL http::/arxxv..og///// 2010.02180 arXiv: 2010.02180.
Steven Pinker. How the mind works. Norton, 1997.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: General- ization Beyond Overfitting Small Algorithmic Datasets. arXiv:2201.02177 [cs], January 2022. URL http:///aaaaw..ooaaa/////////
arXiv: 2201.02177.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023,
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability, 2017.
Omkar Ranadive, Nikhil Thakurdesai, Ari S. Morcos, Matthew L Leavitt, and Stephane Deny, On the special role of class-selective neurons in early training. Transactions Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https:::p//............t http:////oeereeee.eeeeeeee fori//oorooo forum? ? d=
JaN1H6dZYK
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring.
In Pruceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2699-2712, Online, July 2020. Association for Computational Linguistics. doi: 18.18v53/v1/
2020.acl-main.244 240. URLlptttps.//allennooooooyg www.:tthooolyyo.oggg/gg wat..rrg/2020.ac1-mmr -ma..cc4mmm.m.2
Naomi Saphra. Interpretability Creationism. The Gradient, July 2023. URL
https://
Naomi Saphra and Adam Lopez. Understanding Learning Dynamics Of Language Models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 3257-3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N0.18653/v1/N111 1329. URL/attps://wwaat httptttppss/www.aallleeeett www.wwrwwbebrro.ooo.o.oo.ooo wwwprr//nnnooooo///oooooo 1329
Naomi Saphra and Adam Lopez. LSTMs compose and Learn Bottom-up. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2797-2809, Online, November
2020. Association for Computational Linguisties. doi: 10118655/1120200 wtt.://........n..s.....ma moddings-emnlp.22 2552.
URL https::p//// /ttttt:///ccaaaaee http::://acaatttaai o
25.2
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are Emergent Abilities of Large Lan-
guage Models Mirage?, April 2023. URL
arXiv:2304. 15004 [es].
Sebastian Schuster and Christopher D. Manning. Enhanced English Universal Dependencies: An improved representation for natural language understanding tasks. Proceedings of the Tenth International Conference Language Resources and Evaluation (LREC' pp. 2371-2378, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https
//aclaanthology..rrgLL----
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. The multBERTT: BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/v 1/P.6 1162. URL https://ecc antho http://accclattooooooooooooooo-. logy.org//16-1115
14

---

Sofia Serrano and Noah A. Smith. Is Attention Interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computationnl Linguistics, pp. 2931-29511 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 11186633/1/11991111 URL https: //aclantho hww.llannnoooy.og//PPP... ogy.orgg/l....oooooooog 9- 9-1122
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The Pitfalls of Simplicity Bias in Neural Networks. arXiv:2006.077 10 fcs, star), October 2020. URL http://arxiv.orgg /btpaa/.//.00000
arXiv: 2006.07710.
Ravid Shwartz-Zii Information flow in deep neural networks. arXiv preprint arXIv:2202.06749, 2022.
Ravid Shwartz- Ziv and Naftali Tishby. Opening the Black Box of Deep Neural Networks via Infor- mation. arXiv: 1703vv1703.000 300100 [cs], March 2017a. ULL/itte//aaxxxoor///////1333330 arXiv: 1703.00810.
Ravid Shwartz-ZI Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100.:0003011111 2017b.
Ravid Shwartz-Za Zivi Amichai Painsky, and Naftali Tishby. Representation compression and general- ization in deep neural networks, 2018.
Aarohi Srivastava et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, June 2022. URL http:://araat.rrr/aa//20.......0 Number: arXiv:2206.046 05 (.Xii:22060046111 [cs, stat].
Ilya Sutskever.
An observation on Generalization, August 2023.
URL
https:/
www.yout www.yyyuuube.ommm/ www.w///www.w//aa/ooooo AKMuA TVZ3A
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13:12, 2019.
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon, June 2022. URL |htt:: http::://axvvvooo///bbb. /ttp://w..rrr/bbb//2200044333330
Number: arXiv:2206.04817
arXiv:2206.04811 [cs, math].
Mycal Tucker, Peng Qian, and Roger Levy. What if This Modified That? Syntactic Interventions via Counterfactual Embeddings. September 2021. UKL/nttp.//arrvvorr///////.....1 arXiv:2105,14002 [cs].
Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-ffunction map is biased towards simple functions, April 2019. URL http:// arXiv:1805.08555 805.0852 [cs, stat].
arrxiw.org/abs/1105..0522
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias, 2020. URL
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. arXiv preprint arXiv:1905.00478, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. wwtgg//w8---- 5446
doi: 10. 10.18653/v1/W18 1446..
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng- Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377-392, 2020a. doi: 10. 1662/cc/,,,, _aaddlla_________ 00321.
URL httpss///oo..org/1...1.....
00021
15

---

Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. Learning Which Features Matter: RoBERTa Acquires Preference for Linguistic Generalizations (Eventually). In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 217-235, Online, November 2020b. Association for Computational Linguistics. URL https: www.ac//webbrrg//atthhooo.oooo
n.1 6
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Mod- els. June 2022. URL arXiv:2206.07682 [cs].
Number: arXiv:2206.07682
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020, Association for Computational Linguistics. Stt.:tooo////ww..ueeeeeeee/ortoo
demos.6
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When Do Curricula Work? arXiv:0012.03107 [cs, eess, stat], February 2021.
arXiv: 2012.03107.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales, 2023.
Ed Yong. We're the Only Animals With Chins. and No One Knows Why. January 2016. URL /tt.ap/pww//etttaaaaaaaaooooaaaa
Section:
Science.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralaa, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
16

---

A SYNTACTIC ATTENTION STRUCTURE PROBE
We use the following simple probe, based on (Clark et al. 2019), to measure syntactic attention structure. First, we define head-specific probe Jn., that predicts the parent word for a target word (h.10 I from the attention map in head h and layer l. Denoting aig as the attention weight between words and j for attention head h in layer I, we define this probe as:
(h.1) (x:) arg max max (iij z5
.ajs (0.0))
In other words, given target word i and attention head h layer I, Sn. predicts the other word that receives the maximum attention across both directions (c.g.. from parent to child and child to parent). Since BERT Base uses byte-pair tokenization (Sennrich et al.) 2016), we convert the token- level attention maps to word-level attention maps. Attention word is summed over its constituent tokens, and attention from a word is averaged over its tokens, as in Clark et al. (2019).
However, this probe is head-specific. To then acquire parent predictor for a given dependency relation, we select the best rest-perforning head (as determined by accuracy of the predicted parent words when compared against silver labels) for each relation. Denoting the set of all dependency relations as R and each relation R E R as the set of all ordered word pairs (æ;, y) that have that relation (with y being the parent of r), the best-performing probe for each relation R
hR = arg max
Σ
Σ In (I (Ii (Hinsn)(TT)))
R
Ju.1
RED (ZL.Z))ER
where D is the dataset, I2 and 2 represent the i-th and j-th words in example x, and In is the indicator function for set R-i.e., n (( fn,(x))) = if (ri, fn,(2:)) E R (which occurs only when (x,) Ty since each word only have one parent) and 0 otherwise. Lastly, we these relation-specific probes to compute the overall accuracy across all dependency relations. The resulting accuracy is known the Unlabeled Attachment Score (UAS):
UAS ΣNER
Σ Σ
Σ
1s ((r,
(5)
R
RER ZED (N...))ER
We compute UAS on a random sample of 1000 documents from the Wall Street Journal portion of the Penn Treebank (Marcus et al. 1999) and use Stanford Dependencies (Schuster & Manning 2016) parses as our silver labels of which word pairs (Ii,Tz) correspond to each dependency relation RER R.
B SYNTACTIC REGULARIZER
We add syntactic regularizer that manipulates the structure of the attention distributions. The regularizer adds a syntacticity score ((i, xrj) that is equal to the maximum attention weight (summed across the forward and backward directions) between words i and j, for all pairs (i, where there exists some dependency relation between i and j. Because heads tend to specialize in particular syntactic relations, we compute this maximum over all heads and layers. More precisely,
((,,, max (h.f) h,l
max 0 (h.1) h,
ij
ji
We use this regularizer to either penalize or reward higher attention weights on a token's 's syntactic neighbors by adding it to the MLM loss LMIM. scaled by constant coefficient A. We set À < 0 and > to promote and suppress syntacticity, respectively. If we denote parent as the dependency parent of x and ((x) = {vl par)nt(y)) as the set of all dependents of cr, then the entire loss objective is:
[(x) [M +
Σ
(7)
Original loss
((E)(
Syntactic regularization
17

---

C
RELATED WORK
Our work links two parallel bodies of literature, one from the interpretability community, focusing on causal methods of interpretation: and the other from the training dynamics community, focusing on phase changes and on the influence of early training. We combine insights from both communities, studying the role of interpretable artifacts on model generalization by measuring both variables while intervening training.
C.1 SIMPLICITY BIAS AND PHASE CHANGES
Models tend to learn simpler functions earlier in training (Hermann & Lampinen, 2020 Shah et al., 2020:
Nakkiran et al. 2019; Valle-Péres et al., 2019 Arpit et al. (017)) 7). In LMs. Choshen al. (2022) identify trend in BLiMP (Warstadt et al. 2020a) grammatical tests during training: earlier on, LMss behave like n-gram language models, but later in training they diverge. Likewise, LMs learn early representations that are similar to representations learned for simplified versions of the language modeling task, like part-of-speech prediction (Saphra & Lopez, 2019). Despite gradual increases in complexity, SGD exhibits a bias towards simpler functions and features that are already learned (Pezeshki et al., (021). so simplistic functions learned early in training can still shape the decisions of a fully trained model. Importantly, large degree of simplicity bias can be disadvantageous to robustness, calibration, and accuracy (Shah et al. 2020), which inspires our approach of limiting access to interpretable- and thus simplistic, as interpretable behaviors must be simple enough to understand (Lipton solutions early in training.
In studying transitions between simplistic internal heuristics and more complex model behavior, we incorporate findings from the literature that identifies multiple phases during training (Jastrzebski
al. 2020 Shwartz-Ziv & Tishby| 2017a). While often, the performance of language models scales
predictably (Kaplan et al.) [220]] SSrvasttvaa et al. 2022). some tasks instead show breakthrough behavior where a single point along a scaling curve (i.e. compute/number of tokens seen/number of
parameters versus test loss) shows a sudden improvement in performance (Srivastava et al.) 2022
Wei et al. 2022] Caballero et al. 2023). One computational structure, the induction head, emerges in autoregressive language models a discrete phase change (Olsson et al. (022)) and is associated with handling longer context sizes and in-context learning. In machine translation, Dankers et al. (2022)
find a learning progression in which a Transformer first overgeneralizes the literal interpretations of idioms and then memorizes idiomatic behavior. When the training set makes grammatical rules ambiguous, Murty al. (2023) show that language modelling eventually leads phase transitions towards the hierarchical version of a rule over an alternative based on linear sequential order. Outside of NLP, phase changes are observed in the acquisition of concepts in strategy games (Lovering et al., 2022; McGrath et al., (202)) and arithmetic (Liu et al.) 2022] Nanda et al., (023). Our work also identifies specific phase in MLM training, the SAS phase, and analyzes its role in performance and generalization behavior.
We also observe an alignment between phase changes in representational complexity and in gen- eralization performance (Section 4.1). which parallels the observation of Thilak et al. (2022) that generalization time during grokking (Power et al. 2022) aligns with the timing of cycles of classifier weight growth. Lewkowycz et al. (2020) and Hu et al. (2023) report a similar alignment between classifier weight growth and the early transition when loss first begins to decline. Our results indicate that, in natural settings, such phase changes in complexity happen intermediate points in training when a model first acquires particular representational strategics.
Finally, although we find the same phase transition occurs across multiple training runs, other work indicates that generalization capabilities are sensitive to random seed (Sellam et al., [022] Juneja et al., 2023 Jordan 2023). Furthermore, phase transitions may be primarily artifact of poor hyperparameter settings (Liu et al.) (003)), which lead to unstable optimization (Hu et al. 2023). Therefore, it is possible that these abrupt breakthroughs would vanish under the correct architecture and optimizer settings.
C.2 INTERPRETING TRAINING
Recent work in interpretability has begun to take advantage of the chronology of training in developing better understanding of models. In some of the first papers explicitly interpreting the training process,
18

---

Raghu et al. (2017) and Morcos al. (2018) subspace methods to understand model convergence and representational similarity. Adapting their methods, |aahhra & Lopez (0199) find that early in training, LSTM language models produce representations similar to other token level tasks, and only begin to model long range context later in training. Liu et al. 2021) use diverse set of probes to observe that RoBERTa achieves high performance most linguistic benchmarks early in pre-training, whereas more complex tasks require longer pre-training time.
Some studies find that specific capabilities are often learned in a particular order. In autoregressive language models, Xia el al. (023) show that training examples tend to be learned in a consistent order independent of model size. In MLMs. Chiang al. (2020) find that different part of speech tags are learned at different rates, while Warstadt et al. (2020b) find that linguistic inductive biases only emerge late in training. Our work likewise finds that extrinsic grammatical capabilities emerge at consistent point in training.
While our phase transition results mirror Murty et al. (2022)'s 's findings that the latent structure of autoregressive language models plateaus in its adherence to formal syntax, their work also finds the structure continues to become more tree-like long after syntacticity plateaus. Their results suggest that continued improvements in performance can still be attributed to interpretable hierarchical latent structure, which may be an inductive bias of some autoregressive model training regimes (Saphra & Lopez )0
Although Appendix I precludes the impact of thresholding effects (Schaeffer et al. 2023 Srivastava et al.] 2022) our results, the relationship between the structure onset and capabilities onset does reflect a dependency pattern similar to the checkmate-in -enecmaat--i--oee task, which Sriiastava et al. (2022) consider to be precipitated by smooth scaling in the ability to produce valid chess moves. Even in cases where there is no clear dependency between extrinsic capabilities, there may be internal structures like SAS that emerge smoothly, which can be interpreted as progress measures (Barak
et al. 2022 Nanda et al. 20233] Merrill et al. (023).
C.2.1 INTERPRETATION THROUGH INTERVENTION
Claims about model interpretations can be subject causal tests, typically applied at inference time (Vig et al., [2200] Meng et al., 2023). Although causal training interventions are rare. some existing work has used them to support claims about interpretable model behavior. [eavitt & Morcos (2020). for example, control the degree of neuron selectivity during training in order to demonstrate that this transparent behavior was often, in fact, maladaptive. Follow-up work (Ranadive et al.) 2023) shows that neuron selectivity is only transiently necessary early in training, implying that selective neurons are ultimately vestigial. Likewise, Olsson et al. (2022) modify the Transformer architecture to mimic induction head circuits, in order to test their claim that induction head formation was responsible for an early phase change. In considering the role of SAS in model performance, we likewise intervene during training to support and suppress this behavior.
Our work closely relates to the literature on critical learning periods (Achille et al. 2018), where biased data samples prevent the acquisition of particular features other model behaviors early in training, leading to a finding that a model which fails to learn certain features early in training cannot casily acquire them later. While those experiments illustrate different phases by removing early features and damaging performance, our experiments elicit positive changes by removing certain early behaviors, in order to promote other strategies. Furthermore, the behaviors we suppress early in training are immediately learned as soon as they are permitted, so these phases would not be considered critical learning periods.
In all the preceding experimental work that infers causal relationships, it is possible that some related factor is also affected by the proposed intervention. Our work must likewise confront the possibility of an entangled factor that responds to our intervention. The standard approach to remedy this issue is to intervene in as targeted a way as possible, ensuring minimal entanglement between the targeted factor and other factors. Since our approach specifically targets internal syntax structure, we expect that the causal relationships we infer are direct result of changes in internal syntax representations. even if these representations not exactly SAS but strongly associated with SAS. We also observe that the capabilities onset consistently appears after the SAS onset, even as we adjust the timing of the SAS onset arbitrary points, which supports the connection between SAS- closely related
19

---

structural pattern and grammatically capabilitiis. To the best of our knowledge, this is novel causal analysis that is enabled by our developmental lens.
D BLIMP IMPLEMENTATION DETAILS
consists of 67 different challenges of 1000 minimal pairs each, covering a variety of syntactic. semantic, and morphological phenomena. To evaluate, we use MLM scoring from Salazar et al. (2020) to compute the pseudoperplexity (PPPL.) of the sentences in each minimal pair, defined in terms of the pseudo-log- lowelloo--iielooodddd (PLL) score:
BLiMP Warstadt al., 2020a)
Σ PLL(2))
PPPL(D) = exp
(8)
RED
where D is a corpus of text and N is the size of D. Additionally, PLL is defined as
PLI()) = Σ log PMLM (z:la,;)))
where PArLM (xilas 0) is the probability assigned by the model parameterized by to token x.. given only the context with the i-th token masked out.
The BLiMP accuracy is computed as the proportion of acceptable sentences assigned higher PI.. (or lower PPPL) than the unacceptable alternatives. For example. consider the minimal pair consisting of the sentences "These patients do respect themselves" and "These patients do respect himself," where the former is linguistically acceptable and the latter is not. Suppose BERT Base assigns the former sentence an average PLL of -0.8, and the latter an average PLL of -6.0. Then we consider BERT Base to be correct in this case, since the average PLL of the acceptable sentence is higher than the PLL of its unacceptable counterpart.
E CORRELATION BETWEEN UAS AND CAPABILITIES
When we consider all 25 MultiBERTs seeds, we can measure the degree to which natural random variation yields correlation between model quality and implicit parse accuracy (UAS) from SAS. We find thhat UAS does not correlate with either the MIM test loss (R2 2821) or the grammatical capabilities measured by BLiMP (R² -317). This complete lack of significant correlation is clear Fig. Therefore, correlational results not support the common assumption that SAS leads to grammatical capabilities.
1.69
0.84
Mean BLIMP Score
Loss
1.63
0.83
0.38
0.40
0.42
0.44
0.46
0.38
0.40
0.42
0.44
0.46
Figure 6: Across 25 MultiBERT's seeds, we do not find a significant correlation between implicit parse accuracy (UAS) and either (a) MLM test loss (R²2 -2821) (b) grammatical capabílities (R2 = -317).
20

---

F MULTIBERTS DEVELOPMENTAL ANALYSIS
MultiBERTs (Sellam et al. 2022 is a public release of 25 BERT-base runs, 5 of which have intermediate checkpoints available. Although the intermediate checkpoints are not granular enough to show the timing of the abrupt spike in implicit parse accuracy, to show clear break in the accuracy curve for BLiMP, the results nonetheless align with ours (Fig. 7 UAS clearly shows a sharp initial spike followed by plateau within 20K timesteps. It appears that the BLiMP increase also occurs within the first 20K steps. The loss drops precipitously initially and more slowly after the 20K step checkpoint, as we would expect from the other metrics.
The timing of the UAS plateau implies a slightly faster timeline for the acquisition of linguistic structure compared to our reproduction, and the loss is slightly lower than higher BLiMP average. Although MultiBERT's appears to be a closer reproduction of BERT with better results compared our run, we find that it nonetheless is compatible with the same phase transition.
well, with slightly
0.4
0.8
BUMP Accuracy
0.3
0.7
0/5
0.2
0.6
0.1
0.5
5 Step
10
10
10
x10
Step
x10
Step
x10
Figure 7: Metrics over the course of training for the 5 MultiBERT's seeds released with intermediate checkpoints. On y-axis: (a) MLM loss (b) Implicit parse accuracy [c) average BL.iMP accuracy, with confidence intervals computed across tasks.
G COMPLEXITY AND COMPRESSION
Interpretable behaviors such as SAS, by nature of their understandability. must be simplistic. Coin- cidentally, models tend to learn simpler functions earlier in training (Hermann & Lampinen 2020
Shah et al. 2020 Nakkiran et al. 2019 Valle- Pérez et al. 2019 Arpit et al. 2017). a tendency often referred simplicity bias. However, too much simplicity bias can be harmful (Shah et al.) 2020)
although simplistic predictors can be parsimonious, we may also lose out on the predictive power of more complex, nuanced features.
we view SAS as an example of simplicity bias, then we can also view this phase transition through an information theoretic lens. The Information Bottleneck (IB) theory of deep learning (Shwartz- Ziv & Tishby| 2017b) states that the generalization capabilities of deep neural networks (DNNs) can be understood a form of representation compression. This theory posits that DNNs achieve generalization by selectively discarding noisy and iark-itrelevant information from the input, while preserving key features (Shwartz-Ziv (002). Subsequent research has provided generalization bounds that support this theory (Shwartz-Ziv et al. 2018 Kawaguchi et al. 2023). Similar principles have been conjectured to explain the capabilities of language models (Chiang 2023 Cho 2023) Sutskever 2023). Current studies distinguish two phases: initial memorization phase followed by protracted representation compression phase (Shwartz-Ze) Ziv & Tishby| [017bb] memorization, SGD explores the multidimensional space of possible solutions. After interpolating, the system undergoes phase transition into a diffusion phase, marked by chaotic behavior and a reduced rate of convergence the network learns to compress information.
et al., 2023). During
To validate this theory in MLM training, we analyze various complexity metrics as proxies for the level of compression (see Fig. 2(a) for TwoNN intrinsic dimension (Facco et al. 2017) and
21

---

Appendix L.2 for additional complexity/nformation metrics). Our results largely agree with the IB theory, showing a prevailing trend toward information compression throughout the MLM training process. However, during the acquisition of SAS, distinct memorization phase emerges. This phase, which begins with the onset of structural complexity, allows the model to expand its capacity for handling new capabilities. A subsequent decline in complexity coincides with the onset of advanced capabilities, thereby confirming the dual-phase nature postulated by the IB theory.
H IS THE PHASE TRANSITION CAUSED BY ABRUPT CHANGES IN STEP SIZE?
A possible alternative hypothesis to viewing breakthrough behavior as conceptual "epiphany" would be that it is an artifact of varying training optimization scales. In other words, there may be some discrete factor in training that causes the optimizer's 's steps to lengthen, artificially compressing the timescale of learning. The step size decays linearly and the phase transition happens well after warmup ends at 10K steps, meaning that abrupt changes in the hyperparameters are unlikely to be the explanation. However, there may be other factors that affect the magnitude confirm that the breakthrough is due to representational structure and not due to change in the scale of optimization, we consider x-axis scales using a variety of measurements of the progress of optimization. Rather than considering the number of discrete time steps, consider the following timescales for the weights at timestep t:
step. To
Weight magnitude. Fig. ((a) uses the Euclidian distance from the zero-valued origin, which is equivalent to the weight & norm or /Tw
Distance from initialization. Fig. 8(b) uses the Euclidian distance from the random initialization,
i.e. from the weights at timestep 0: /l 2.
Optimization path length. In Fig. 8(c) we approximate the distance traveled during optimization by adding together the lengths of each segment between weight updates, Σi= v [[w; Wi- 1 Because not every timestep recorded as a checkpoint, we only offer an approximation of the path length by measuring the distance between the recorded sequential checkpoints.
We confirm the phase transitions occur across x-axis scales, Abrupt phase transitions occur whether we consider training timestep. Euclidian distance from initialization, magnitude of the weights, the path length traveled during optimization.
I IS THE PHASE TRANSITION AN ARTIFACT OF THRESHOLDING?
Apparent breakthrough capabilities often become linear when measured with continuous metrics instead of discontinuous ones like accuracy (Srivastava et al. 2022 Schaeffer al. 2023). Is this the case with our accuracy metrics on SAS and BLiMP?
To the contrary, we find that using continuous alternative the accuracy metric shows similar results. In the case of SAS, providing the attention value placed on the correct target, rather than accuracy based on whether attention is highest for that token, gives a continuous alternative to UAS. In the case of BLiMP, we give the relative probability given to the CLS token on the correct answer in the sequence pair, p. compared to the probability P of the incorrect CLS token. In other words, the continuous measurement of BLiMP performance is given by p-g. In either case, we see that the phase transition remains clear (Fig. 9
J GLUE TASK ANALYSIS
Fig. 10 shows the GLUE task breakdown while training BERT Buse. While not all tasks show breakthrough in accuracy at the structure onset, most do.
Most GLUE tasks, meanwhile, do not show marked improvements after brief early stage suppression of SAS (Fig. IT]. The tasks that have more stability across finetuning seeds show a marked decline in performance continue to suppress SH past the alternative strategy onset.
22

---

0.4
Accuracy
8
0.3
Loss
UAS
6
0.2
4
Mean
0.1
0.5
2
250
300
350
250
300
350
250
300
350
|
|6:|
[0:|
(a) Training scale: Euclidian distance of parameter settings from the zero-valued origin, v //wx/la
10
0.8
Mean BL LiMP Accuracy
0.3
0.7
Loss
6
0.2
4
2
|0: (8:-dul: Gols
10:-8ul2
[8
8,12
(b) Training scale: Euclidian distance of parameter settings w2 from the model's random initialization,
Wp
10
0.8
BL LIMP Accuracy
8
0.3
0.7
6
0.2
0.6
4
0.5
2
Oplimization Path
Optimization Path
Optimization Path
Length
Length
Length
(c) Training seale: Total length the optimization trajectory after initialization, Σ given only at checkpoint intervals.
wi 2 with
Figure 8: Learning trajectories of baseline BERT training, with x-axes reflecting various scales for determining the length of training for checkpoint parameters wy at time counting discrete optimization steps. Each curve represents one of three random seeds. Each y-axis corresponds to, from left to right: MLM loss; implicit parse accuracy; and BLiMP average. Structure onset and capabilities onset are both marked on each line.
alternative to
K BLIMP ANALYSIS
As seen in Fig. 12 most BLiMP tasks show similar responses multistage SAS regularization: dip in accuracy for the models that have their regularizer released at the alternative strategy onset and maximum accuracy for model where the regularizer is released after brief suppression. The model released at the alternative strategy onset has the poorest performance for all tasks except ellipsis.
We note that while training BERT for most BLiMP tasks clear improvement occurs at the capabilities onset, though intriguingly, for some tasks there is a decline in performance at the structure onset (Fig. 13
23

---

Average maximum attention weight
0.35
0.6
Relative Likelihood
0.25
0.15
0.05
0.5
Step
x10*
Step
x10*
(a) Averaged maximum attention woight on syntactic neighbors, continuous alternative to UAS accuracy.
(b) Relative likelihood given to the correct member of minimal pair, continuous alternative to BLiMP accuracy.
Figure 9: Continuous, non-thresholded metrics for SAS and BLiMP, across three seeds.
L COMPLEXITY METRICS
The literature model complexity provides abundance of metrics which can provide radically different rankings between models (Pimentel et al., (2020). We consider some common complexity metrics during BERT training, primarily focusing on intrinsic dimension.
L.1 INTRINSIC DIMENSION
In order to measure the complexity of the model and its representations, use Two-NN intrinsic dimension (Facco al., 2017). with other common complexity metrics in Appendix [.2] Two-NN is fractal measure of intrinsic dimension (ID) that estimates the ID d by computing the rate at which the number data points within a neighborhood of radius grows. If we assume that each of points in the d-dimensional ball has locally uniform density, then d can be estimated as function of the cumulative density of the ratio distances to the two nearest neighbors of each data point. Two-NN has been used to study the ID of neural network data representations, and sometimes identify geometric properties that are otherwise obscured by linear dimensionality estimates (Ansuini al. 2019). In our analyses compute the Two-NN intrinsic dimension the [CLS] embeddings of our trained BERT Base models, using pair-wise cosine similarity as our distance metric. We also present the dynamics of several complexity metrics such the empirical Fisher (EF) and weight norm.
L.2 OTHER COMPLEXITY METRICS
Weight magnitude: The norm of the classifier weights an often-studied (Thilak al. 2022 Lewkowycz et al. 2020) metric for model complexity during training. Shown in Fig. 14(a) this metric rises throughout training, with an inflection up at the capabilities onset. Note that this metric is equivalentt to an x-axis scale used in Appendix H
Fisher Information: Inspired by the approach of Achille et al. (2018) we approximate Fisher Information by VLLLMM the trend for which shown in Fig. 14(b) Similar to TwoNN, the model experiences sharp increase in complexity during SAS acquisition, marking the memorization phase. This phase ends abruptly at the capabilities onset, after which slow decrease in complexity characterizes the compression phase and improved generalization.
24

---

COLA
MNLI
MRPC
Matthews correlation
0.8
F1
0.4
0.85
0.75
0.2
0.8
0.7
Accuracy
Accuracy
QNLI
QQP
RTE
0.9
0.57
Accuracy
0.85
0.8
0.55
0.82
0.53
0.7
0.8
0.5
Accuracy
0.78
SST2
STSB
WNLI
0.9
Accuracy
0.8
Accuracy
0.6
0.4
0.85
0.4
0.2
Pearson
0.2
0
5
10 x10*
0
5 Step
10
0
5 Step
10
Step
x10
x10*
Figure 10: GLUE performance across training for BERT Base broken down by task. Structure onset and capabilities onset marked.
M WHAT IS THE ALTERNATIVE STRATEGY?
Thus far, we refer to the acquisition of some competing opaque behaviors, defining them only as the useful behaviors not supported by SAS. We argue that an alternative strategy is being learned in the absence of SAS, and characterize it as the use of long range semantic content rather than local syntactic structure.
The first piece of evidence for the use long-range context is that the onset of the alternative strategy's 's break in the loss curve coincides with the start of increase in performance for longer n-gram contexts on the task of predicting masked word within a fixed length context (Fig. 15(b). For 1000 documents randomly sampled from our validation dataset, randomly select a segment of n + tokens from each document and mask a randomly selected token in each segment. We then compute the average likelihood that the model assigns the masked token. For example, and the randomly selected segment is "a b e d," then we might input the sequence " "[CLS] [MASK] d [SEP]" to the model and compute the likelihood that the model assigns to the token "b' at index in the sequence. As training continues, small contexts while suppressing SAS, over small window.
spikes in performance for increasingly
When training BERT Base. we also (Fig. 15(a) breakthrough in modeling 7-gram contexts across lengths during the structure onset. However, we cannot assess whether a similar set of consecutive phase changes occurs in BERT Base. the difference in timing may occur smaller scale than the frequency of saved checkpoints. all phase transitions in BERT Buse are simultaneous or close to simultaneous, the gradual acquisition of increasingly local structure in BERT'SS SAs. may account for
25

---

COLA
MNLI
MRPC
Matthews comrelaiin
Accuracy
0.88
F1
0.4
0.8
Accuracy mm
0.86
0.78
0.84
0.2
0.82
0.76
QNLI
QQP
RTE
Accuracy
F1
Accuracy
0.57
0.86
0.88
0.55
0.86
0.85
0.53
0.5
SST2
STSB
WNLI
Accuracy
Pearson
Accuracy
0.9
0.85
0.5
0.82
0.88
0.4
0.8
0.3
0.86
0.78
2
4 x10
x10
2
4 x10
Max. no. of steps suppressed
Max. no. of steps suppressed
Max. no. of steps suppressed
Figure GLUE performance for multi-stage regularized models after 100K timesteps, function of the number of steps suppressed and broken down by task. Vertical line marks the BERT alternative strategy onset.
the more gradual onset of the accompanying loss drop (Fig. 3(a). The noteworthy difference that we can confirm is that BERT shows faster initial increase in performance on n-gram modeling compared BERT Base. suggesting that its break in the loss curve relates unstructured n-gram modeling, particularly using long range context.
Another piece of evidence is found in the attention distribution. The attention distribution for BERT is less predictable based on the relative position of the target word (Fig. [6(a)). so unlike BERT the nearest words no longer take the bulk of attention weight. However, on any given sample, the attention distribution is actually higher-entropy for BERTsAS. than BERT Base (Fig. [6(b) indicating that a small number of tokens still retain the model's s focus, although they are not necessarily the nearest tokens. Therefore, BERT SAS- may rely other semantic factors, and not position, to determine where to attend. Note that this evidence is weakened by the fact that attention cannot by directly applied
importance metric (Ethayarajh & Jurafsky 2021)
M.1
ONE BIG BREAKTHROUGH OR MANY SMALL BREAKTHROUGHS?
The loss curve after the alternative phase transition under SAS suppression display appears quite different from the baseline after the SAS onset, because the former trajectory declines far more gradually. A clue to this distinction may be in Fig. 15(a) where we see BERT Base exhibit simultaneous breakthroughs in n-gram modeling at every context length. In contrast, BERT SAS. is characterized by a sequence of consecutive breakthroughs starting from the longest context length and gradually reflecting more local context (although this may account for the difference in transitions the
26

---

Island Effects
Anaphor Agreement
S-sellction
0.6
0.9
0.5
0.7
Argument Structure
Determiner Noun Agreement
Subject Verb Agreement
0.8
0.7
6.0
0.6
Ellipsis
Control Raising
Quantiflers
0.9
0.8
8.7
0.4
Irregular Forms
Npi Licensing
Binding
1.0
0.8
0.8
0.6
0.8
0.6
Max. steps sup pressed
Max. steps suppressed
Filler Dependency
0.8
0.6
Max. of steps suppressed
Figure 12: BLiMP accuracy for multistage models at 100k timesteps, broken down by task. Vertical line marks the alternative strategy onset during BERTSAS- training.
27

---

Island Effects
Anaphor Agreement
S S-selection
0.6
0.5
0.7
0.4
Argument Structure
Determiner Noun Agreement
Subject Verb Agreement
0.7
0.8
0.6
0.6
0.6
Ellipsis
Control Raising
Quantifiers
0.8
0.7
0.6
0.6
0.4
Irregular Forms
Npi Licensing
Binding
0.8
w/4
0.4
5
10 xte
5 Step
10 x7pf
Step
Filler Dependency
0.6
0.4
Step
Figure 13: BLiMP accuracy during BERT Base training broken down by task. Structure and capabilities onsets are marked,
28

---

1.4 X10
10a
Squared Weight Norm
1.2
Fisher Information
1.0
5 Step
10
5 Step
10 x10*
x10
(a) Weight norm.
(b) Approximate Fisher Information.
Figure 14: Complexity metries over time for BERT Base- Structure and capabilities onsets are marked.
structure onset in BERT Base happens later in training when the checkpoints are sampled less frequently, and this may account for the apparent simultaneity). Although the performance at the phrase level shows clear phase transitions, the i.i.d. validation loss appears smooth and gradual from the start of the alternative phase transition. This observation suggestive that smooth i.i.d. loss curve can elide many phase transitions under various distribution shifts, possibly reflecting the conjecture of Nanda Lieberum 2022) that "phase transitions are everywhere."
N EARLY-SUPPRESSION TRAINING CURVES
Fig. 17 illustrates the general trend that, when briefly suppress SAS early in training, recover and even augment the corresponding UAS spike and loss drop. As we continue to suppress SAS, lose these benefits and further weaken the transition to SAS, The best timing for hyper- parameter release is (ERT(38) approach accelerates and arguments the structure onset and improves model quality during the first 100K steps.
SAS. and the training dynamics in Fig. 18 confirm that the multistage
O CONTROLLING FOR TIME ALLOWED TO ACQUIRE SAS
Here, present the same results given in Fig. while controlling for the length of time model trains after releasing the regularizer in multistage setting, instead of total training time. We therefore measure performance exactly 50K steps after setting À to O, instead of measuring 100K steps overall. This allows a shorter overall training time which varies across the models. Therefore, the loss of models with shorter first stages improves less compared the models with longer first stages, Otherwise, the overall patterns remain consistent with the results at fixed 100K time steps.
P LONG TERM IMPACT OF MULTISTAGE SUPPRESSION
To investigate whether the models eventually converge in their biases and structures, look functional differences in the form of total variation distance (TVD, or the maximum difference in probabilities that two distributions can assign to the same event) between the output distributions and representational similarity in the form of centered kernel alignment (CKA; Kornblith et al.) 2019).
Although the models initially become more similar in their output functions, their distance eventually stabilizes with the average TVD between BERT Base and (ER) SAS- falling above the average between pairs of BERT Base seeds, suggesting that the absence of another phase transition. the models will remain distinct their behavior. Meanwhile, the average CKA(BERT Base> (ERT(3)) similarity diverges during the structure and capabilities onsets but converges again towards the average CKA
29

---

Average likelihood of target token
0.12
10
Length of context
6 c
0.08
0.04
4
-0.01
2
0.5
1.5 Step
2.5
X104
(a) Average BERT Base model likelihood of target token, with varying lengths n of unmasked tokens in its immediate context. Line of triangles indicates BERTBaa 's structure onset.
Average likelihood of target token
0.18
10
Length of context
8
0.11
6
0.05
4
-0.01
2
0.5
1.5
2.5 x10
Step
(b) Average SERT model likelihood of target token, with varying lengths of unmasked tokens in its immediate context. Dotted line indicates the alternative strategy onset.
Figure 15: The alternative strategy onset, i.e., the break in loss for BERTsAS. is associated with improvements in n-gram modeling with longer-range contexts. Meanwhile, the structure onset for BERT Base is associated with an improvement in modeling phrases, possibly simultaneously, for all lengths.
BERT Base
BERT Base
Average attention at 100K steps
0.06
BERT SAS.
Average attention entropy
BERTSSAS
0.03
0.00
8
Distance from target token
Step
x10*
(a) Average attention placed on the target token, as function of distance (in tokens) from the target token.
(b) Entropy of the attention distribution, averaged across heads and samples.
The alternative strategy in BERT is associated with sparser attention compared to the attention distributions in BERT Buse However, the average attention of BERT SAS. (as function of position) is overall low, indicating that BERT does not focus attention on nearby tokens as much as BERT Rase does, despite the lower entropy.
Figure 16:
30

---

45
0,8
10
080559 rodivs 50993 po 0. S00
Mean BUMP Accurascy
0,
30
UAS
25
0.6
15
10
x10
Step
x10"
x10"
No suporesiion
With suppression
Figure 17: Metrics over the course of training for multistage SAS-regularized models stopped at various points. On y-axis: (a) MLM loss (b) Implicit parse accuracy (c) average BLiMP accuracy. For all multistage training runs, visualized curves begin only after the regularizer is released, i.c., suppress SAS for the first 10k steps, the curve begins at 10k. Curve for BERT Base is presented as a solid line, with all suppressed models using dashed lines.
45
40
35
Mcon BLMP Accurscy
6.7
15
ae-
180
10
5
10 x10
10 x1o
Step
X10
Step
Step
(a) MLM validation loss.
(b) Implicit parse accuracy.
(c) Mean BLiMP challenge accu-
racy.
Figure 18: Briefly suppressing SAS results in improvements of MLM loss, implicit parse accuracy, and linguistic capabilities early on training.
between different BERT Base seeds later This suggests that even if BERT Base and BERT (3k) high representational similarity, their outputs may still have noticeable differences.
SAS- have
Ultimately, in the long run of training, the difference in quality between BERT SAS- and BERT Basc ceases be statistically significant (Tables [ [. While avoiding SAS early in training accelerates convergence and leads to some functional differences in the final models, the critical learning period (Achille et al. 2018) we observe does not continue to hold at scale.
31

---

at Sox Steps beter Releast
UAS at 50c Steps After Release
40
2.3
35
2.2
30
25
2.1
20
2.0
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
Step suppressed until
Step suppressed until
GLUE at 50c Steps Atter Releas
50K Steps After Releas
0.74
0.70
0.70
0.65
BLMMP at SOK
0.66
10000
2 70000 30000
40000
50000
10000
20000
30000
40000
50000
Step suppressed until
Step suppressed until
Figure 19: Metrics for the checkpoint 50K steps after the regularizer is removed. X-axis is timestep when regularizer with À = 0.001 is removed. On y-axis:
(a) MLM loss shown with standard error of
the mean across batches (b) Implicit parse accuracy (c) GLUE average (d) BLiMP average. Vertical
line marks BERTSAS. alternative strategy onset.
Table 2: Evaluation metries, with standard error, after training for 300K steps (~ 39M tokens), averaged across three random seeds for each regularizer setting. We selected BERT SAS (3k) multistage hyperparameter setting based on MLM test loss 100K steps. We selected 300K the checkpoint to evaluate longer term performance on because longer runs often destabilized, requiring restarts or re-quantization, which force artificial phase transitions late in training.
as the best
MLM Loss
GLUE average t
BLiMP average
BERT
1.55 + 0.00
0.74
0.73 + 0.05 0.77 0.01
BERT SAS+
2.17 = 0.04
0.63 0.04
BERT SAS.
1.76 0.02
0.73
0.62 + 0.05
BERT
(3k)
1.55 0.01
0.76 + 0.02
SAS-
1.0
BERT
CKA versus BERTIa Jaa
BERT
0.5
BERT
(81 BERT
0.9
TVD vs. BERT
#0.4
0.3
0.2
0.8
0.1
0.7
Step
x10*
Step
x10
(a) CKA similarity for the activations of BERT Rase and (ERT(33) (3k)
Average divergence between output distributions of BERT and (ERT(a)) divergence between different of BERT Base-
SAS- compared to the average CKA similarity between different runs BERT Base-
SAS compared the average
Figure 20: Representational and functional similarity over time for BERT Base and BERT (3k) ture (A) and capabilities onsets are marked on each line. Shaded regions are 95% confidence intervals over different pairs of models.
SAS.. Struc-
32


Reviews:
[{"review_id": "sWqYlQ1CP1", "reviewer": "Reviewer_qPGn", "summary": "This paper provides a detailed _developmental_ account of (masked) language models' acquisition of grammatical abilities: the authors measure the the time at which (i) attention heads which pay attention to syntactic structure (dependencies, following prior work showing they emerge) and (ii) grammatical abilities (performance on BLiMP, a linguistic diagnostic set) emerge during training.  In particular, they find that (i) occurs reliably just prior to (ii), though both are abrupt, and that (i) occurs with a _sudden drop in the MLM loss_.  This suggests that the model reliably acquires a certain bit of important latent knowledge (of dependency structure) before behavioral evidence of a skill that uses that knowledge (e.g. grammaticality judgments).  The paper also explores regularization to promote or demote (i), with many, many interesting results about when the sudden drop in loss occurs and how it connects to downstream performance.  This kind of causal intervention shows that MLMs _do_ in fact use syntactic attention heads both when doing masked language modeling and grammaticality judgments, teaching us much more about these phenomena than existing probing methods based on static model artifacts.", "questions": "- Fig 1b: why do you think there's so much more variance in the BLiMP results than in the loss curves and UAS scores?\n\n- I'm curious about whether it matters that silver dependencies were used in regularization.  Did you try any other \"data-free\" regularizers to see if they impact SAS similarly?  E.g. since each token has one head, a regularizer that promotes sparsity of attention should implicitly promote SAS as well and vice versa.\n\n- Missing references: (i) Liu et al 2021, \"Probing Across Time\": https://aclanthology.org/2021.findings-emnlp.71/ .  (ii) p 4: \"Causal methods...\" I have an idea of what works the authors have in mind, but think they should be explicitly cited here.\n\n- Will code and data be made publicly available?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Very detailed analysis of the emergence of certain knowledge and skills _across training time_ in a language model.\n- Demonstrates that drops in loss correspond to acquisition of syntactic knowledge, which then translates to grammatical performance.\n- Methodologically and technically innovative (e.g. regularization as a causal intervention) in a way that moves the state of the probing field forward.\n- Extremely wide range of experiments, helping isolate exactly which features of training and measurement matter for this phenomena.  Crucially, they show that these emergence phenomena are not measurement artifacts, since they persist when a discrete scale (training time) is replaced with several continuous ones.", "weaknesses": "- All of the results are on a single model architecture (BERT base).  On the one hand, this makes sense, since an extremely wide range of experiments are carried out.  On the other hand, we don't know whether the connection between sudden drops in the loss and syntactic knowledge would apply at larger scales, with causal language modeling, etc.\n- There are so many experiments and interesting observations that the main paper makes very frequent reference to a plethora of appendices for more detail.  This makes it a bit hard in places to figure out _exactly what_ is being reported and what it all means.  (E.g. the discussion of the Information Bottleneck was fairly hard to follow, even to someone who knows a bit about that literature.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEvAsnvxBM", "reviewer": "Reviewer_cVcx", "summary": "This paper analyzes sudden transitions in the training loss of BERT models, identifying two components to this drop: the development of attention patterns correlated with syntax (SAS), and the subsequent emergence of the ability to make grammaticality judgements. The paper then manipulates SAS via a additional term in the loss and analyzes the effect of these manipulations on the second component. The findings are twofold: (i) acquisition of SAS is a pre-requisite for the grammatical capabilities and (ii) briefly supressing SAS leads to a subsequent increase in grammatical capabilities.", "questions": "Do I understood correctly you equate the syntax-like attention patterns with simplicity bias, and at some point call them \"simple heuristics\" (section 5.1)?\nThis is a bit confusing as in the NLP literature terminology like \"simple heuristics\" refers to undesirable reliance on surface lexical patterns (like bigrams), and reliance on syntax is considered the opposite of a simple heuristic. It would be good to make sure your unusual framing is not a cause of confusion to the readers.\n\nMinor doubt: since you use WSJ data for testing, why use silver Stanford parser dependencies instead of gold, converted from the manually created trees?\n\nDo you have any inkling of what your mystery alternative strategy may involve?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is one of relatively few papers which analyzes learning dynamics in BERT and the transitions identified are intriguing. The paper has the potential of stimulating more work in the same vein, applied to models more advanced than BERT.", "weaknesses": "I did not find any major weaknesses in the paper. There is a lot going on, but this is understandable given the novelty of the approach. \n\nThat said, some of the framing and terminology could be explicated a bit more carefully. I found the issue of simplicity and simplicity bias especially muddled (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27MPVnFF79", "reviewer": "Reviewer_4MRy", "summary": "The research highlights that understanding model behavior requires observing the training process trajectory, not just analyzing a fully trained model. This study looks into syntax acquisition in masked language models, focusing on the Syntactic Attention Structure (SAS). It shows that SAS emerges suddenly during a specific pretraining phase, marked by a significant loss reduction. Further experiments manipulating SAS confirm its essential role in developing linguistic capabilities, whereas the experiments also find that briefly suppressing SAS improves model quality. The authors explain that SAS competes with other effective traits.", "questions": "- I would like to know what kind of method/technique is used for encoding positional information. Is it the same as the original positional encoding in Vaswani et al. 2017 (https://arxiv.org/abs/1706.03762) or something more recent variants such as Rotary Positional Encoding (https://arxiv.org/abs/2104.09864v4)? I'm asking this because the syntactic dependency relates to positional information in the sentence. I wonder how much it affects (or does not affect) the experimental setup in this paper.\n\n- This is a more open-ended question but I wonder whether we can observe similar breakthrough (steep drop in loss) in auto-regressive (causal / decoder-only) LMs.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- The research idea and findings (including the appendices) are both intriguing and worthy of being shared with the community.\n- The experiments were conducted and executed effectively.", "weaknesses": "- I didn't find any major weaknesses, just a few minor questions (detailed below).\n- Some individuals might express concerns that the experimental setup is somewhat minimal and may suggest the inclusion of additional elements, such as utilizing RoBERTa or evaluating the model on other, possibly more recent, benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4GN7UKjBj", "reviewer": "Reviewer_uEHm", "summary": "This paper monitors masked language models’ development of syntactic attention structure (SAS) in the attention pattern, grammar capability (measured by the BLiMP dataset) and the GLUE score. They find that\n\n- The model’s grammar capability spikes right after the spike of the model’s SAS score.\n- They measure the complexity of the model throughout the pretraining process, and claim that the trend is aligned with the Information Bottleneck Theorem.\n\nThey also use a “regularization loss” to interfere with the acquisition of SAS. They find that enhancing/suppressing SAS improves/harms the grammar capabilities of the model. \n\nThough the model is able to develop an “alternative strategy” for acquiring the grammar capabilities when SAS is suppressed, they find that lifting the suppression of SAS during the “phase transition” leads to worse grammar capabilities, while lifting the suppression before the “phase transition” can result in better grammar capabilities. They discuss this phenomenon as related to the simplicity bias of models.", "questions": "## Q1: About section 4.1.1\n\nThe authors discuss their findings along with the information bottleneck (IB) theory. \n\n1. It’s unclear to me how the findings agree with what part of the IB theory.\n2. It’s also unclear to me how this is related to the findings or the arguments in this paper.\n\n## Q2:  The specific meaning of phase transition in Sec 4.2\n\nIn Sec 4.2, the term “phase transition”. Could you clarify what it refers to? Does it refer to the period between the structure onset and the capabilities onset.\n\n\n## Q3:  The importance of understanding phase transition in general\n\nI understand that *phase transition* is a *hot topic* for some model interpretability community. However, in this work, could you provide more context in which studying *phase transition* is important?\n\n## Suggestions\n\nI understand that every paper needs a reasonable scope to work on and I don’t expect that one single paper explains everything. However, I would suggest that the authors scope this paper more explicitly. “Emergence”, “phase transition” and “capability”, for example, I think are some very general terms, and this paper focuses only on some specific aspects of them. Scoping more clearly and explicitly in the introduction section will help readers (at least me) understand this paper more easily, especially when this paper is discussing MLM models while these terms are usually co-occur with autoregressive language models.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. They empirically show the close relationship between the spike of SAS score, grammar capability, and GLUE score. This is interesting because it may validate the role of understanding syntactic structure for downstream tasks.\n2. This work provides many experimental results, which could be useful for better understanding of the model pretraining dynamics.", "weaknesses": "# Main concerns:\n## 1. The causal relationship between SAS and BLiMP/GLUE scores\n\nI think even though the spike of the BLiMP/GLUE score follows closely after the spike of the SAS score, it is not well substantiated to say that SAS is necessary for the capabilities required for BLiMP and GLUE. The intervention experiment in Sec 4.2, 4.3 can not support the causal relationship between them either. If there is a latent factor X that causes the better SAS, BLiMP, GLUE scores, adding the regularization term may suppress that latent factor X in addition to the SAS score. In this case, suppressing SAS also leads to worse BLiMP and GLUE scores. So that SAS is necessary for BLiMP and GLUE is not the only explanation for the observation in Sec 4.3, 4.3.\n\n\n## 2. The arguments about the simplicity bias is not clear (to me)\n\nIt seems that this paper suggests that SAS indicates that the model suffers from some simplicity bias issue, which is counterintuitive to me. In general I think people use simplicity bias to explain some robustness issues because some spurious (unreliable/non-causal) features are simpler to learn than causal features, or say the model is doing some shortcut learning. However, it is hard to imagine that the syntactic structure is something the model shouldn’t rely on to solve any NLP problem.\n\nI think to talk about simplicity bias, the author should be more clear about the definition of “simplicity” and provide more evidence that the model’s prediction is really “biased” by that specific “simple” feature.\n\n\n## 3. The motivation of the study\n\nIt’s unclear to me why we should look at the development of these “capabilities”. I would like to know how the findings in this paper can potentially direct future research directions?\n\nIn general I feel that it’s cool that this paper uses some fancy techniques to show many findings and defines some interesting terminologies. However, it’s unclear to me what the high-level message of this paper is. Unable to capture the coherent theme of this paper, I found it difficult to put all the information in this paper together.\n\n# Minor issues:\n\n1. This paper should be more specific about the definition of “capabilities”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)