Unified View of Grokking, Double Descent and Emergent Abilities: A Comprehensive Study on Algorithm Task
Authors: Yufei Huang
Shengding Hu
Xu Han
Zhiyuan Liu
Maosong Sun
Venue: COLM (2024)
Keywords: Deep Learning, Grokking, Double Descent, Emergent Abilities
TL;DR: We provide a comprehensive study on algorithm tasks to analyse different training dynamics with various model sizes and training data size and propose a framework to unify grokking, double descent and emergent abilities.

Abstract: Recent studies have uncovered intriguing phenomena in deep learning, such as *grokking*, *double descent*, and *emergent abilities* in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models. In this paper, we present a comprehensive study on algorithm task to provide a unified view of these three phenomena, with a focus on the interplay between memorization and generalization. Through extensive experiments spanning a wide range of model sizes and training data quantities, we uncover four distinct training dynamics, each arising from unique combinations of model size and training data quantity, formulating a theoretical framework for further analysis. Utilizing this framework, we establish connections between *double descent* and *grokking* and propose two verifiable predictions regarding the occurrence of *double descent*, both substantiated by our experimental results. Moreover, we expand our experiments to the multi-task learning paradigm, demonstrating how algorithm tasks can be turned into emergent abilities by mixing some pure memorization data. This offers a novel perspective to understand *emergent abilities* in Large Language Models.

Published as a conference paper at COLM 2024
Unified View of Grokking, Double Descent and Emergent Abilities: A Comprehensive Study on Algorithm Task
Yufei Huang!.23 Shending Hul223, Xu Han Zhiyuan Liu Maosong 123+
DDpartment of Computer Science and Technology, Tsinghua University, Beijing BBeijnng National Research Center For Information Science And Technology
Ilnstitute for Artificial Intelligence, Tsinghua University, Beijing huangyf530@gmail.. com
Abstract
Recent studies have uncovered intriguing phenomena in deep learning, such grokking, double descent, and emergent abilities in large language models, which challenge human intuition and are crucial for a deeper un- derstanding of neural models. In this paper, we present comprehensive study on algorithm task to provide unified view of these three phenomena, with focus on the interplay between memorization and generalization. Through extensive experiments spanning a wide range of model sizes and training data quantities, we uncover four distinct training dynamics, each arising from unique combinations of model size and training data quantity, formulating a theoretical framework for further analysis. Utilizing this framework, we establish connections between double descent and grokking and propose two verifiable predictions regarding the occurrence of double descent, both substantiated by our experimental results. Moreover, we ex- pand our experiments to the multi-task learning paradigm, demonstrating how algorithm tasks can be turned into emergent abilities by mixing some pure memorization data. This offers novel perspective to understand emergent abilities in Large Language Models.
1 Introduction
There are several interesting phenomena Deep Learning, among which grokking (Power et al., 2022), double descent (Belkin et al., 2019; Nakkiran al., 2020) and emergent abili- ties (Brown et al., 2020; Wei et al., 2022a; Ganguli et al., 2022; Srivastava et al., 2023) in current Large Language Models attract a lot of attention. Understanding these phenomena is important for us to reveal the mechanism of deep learning. Plenty of works (Liu et al., 2022; 2023; Thilak et al., 2022; Varma et al., 2023; Schaeffer et al., 2023; Michaud al., 2023) have been done to explain these phenomenons from different perspectives. However, these works all concentrate on a single phenomenon and explain them separately. In this work, provide preliminary study to give unified view of these three phenomena from the perspective of competition between memorization and generalization.
Our work is based on Varma et al. (2023)'s explanation for grokking. They attribute grokking to the competition between two distinct types of circuits in the model: one responsible for memorization, which achieves high training accuracy but poor validation accuracy, and another for generalization, capable of high performance in both training and validation. The latter, although slower to develop, proves more efficient in terms of parameter norms, leading to the model finally transferring from memorization to generalization to achieve higher efficiency. Intriguingly, the efficiency of the memorization circuit is inversely related to the volume of training data, indicating that larger datasets reduce its efficiency. In contrast, the efficiency of the generalization circuit remains consistently stable, regardless of the size of the training data. As a result, Varma et al. (2023) identified critical dataset size, Deritr delineating boundary. Within this boundary, memorization and generalization circuits
Corresponding author.

---

Published as a conference paper at COLM 2024
Memorization Capacity
This Accency
Valdessn Acorpyy
pregreddo
1.4-
pregression
Train Dain Size
wo double descent
40000
(d) grodking
40303
20
40/01
-300
(oj icnn-grrkkhing
double descent
1.2
(o) godkiigg
1.2
podding
Critical Dataset
(b) memorization
Size
28 32
1.1
Model Hidden Size
Egech
The increasing memorization capacity and decreasing critical dataset size for larger models split the figure into four distinct zones including progression, memorization, semi-grokkiing and grokking. Each zone will show a specific training dynamic illustrated in the right side. Some important intersections are marked with estimated values. Right: Each figure represents a specific training dynamic: (a) Progression, demonstrated using a model (b) Memorization, shown with model having a hidden size of 32, trained on 2600 data points. (c) SemiiGrokking, depicted with model of hidden size 56, trained on 2600 data points. (d) Grokking, visualized using model with a larger hidden size of 56, trained on 4000 data points. These figures exemplify the variable training dynamics of models with different configurations to specific training data quantities.
Figure Left:
with hidden size of 8 and trained on 2600 data points.
demonstrate similar levels of efficiency and the model will show part of generalization ability. Crossing this threshold significantly enhances the probability of grokking.
Building upon their findings, our study expands the investigation to various model sizes. We firstly observe that smaller models require larger critical dataset size for grokking, implying an inverse relationship between model size and the necessary amount of training data for grokking. Conversely, model's memorization capacity is directly proportional to its size, indicating that smaller models have reduced capacity for memorization. We confirm these relationships by experiments in 5 2.
The reverse relationships with model size inevitably lead to an intersection point of the two curves (yellow star in Figure 1). The two curves with this intersection point create four distinct zones on the graph, each reflecting a unique training dynamic in our experiments as shown in the right side of Figure/ 1. (a) Progression: When the training data size is beyond model's memorization capacity, model will be unable to memorize all of them, causing model first memorizing as much training data as possible with a zero validation performance and then generalizing to part of validation data with an increase in train accuracy at the same time. (b) Memorization: For small amount of training data, model has the ability to memorize all of them and the memorization circuits is more efficiency than generalization. Therefore, model will only memorize the training data with zero validation performance. (c) Semi- Grokking: When the number of training data points approximates the critical dataset size, the model exhibits moderate generalization capabilities after memorizing all the training data. This behaviour was first identified by Varma et al. (2023). (d) Grokking (Power et al., 2022): When the number of training data is beyond critical dataset size, the generalization circuits become more efficiency than memorization, leading the model transfer from memorization to generalization long after training performance become perfect.
Analyzing Figure 1, we can easily link double descent with grokking and figure out when double descent will happen. Specifically, for training data volumes below the intersection point, the model undergoes progression, memorization, semi-grokking and then grokking with model size increasing, leading the validation performance first increases, then decreases, and ultimately increases again, which is exactly the double descent phenomenon (Belkin et al., 2019; Nakkiran et al., 2020). In contrast, when the quantity of training data surpasses the intersection point, an increase in model size results in the model through progression to grokking, resulting in consistently positive correlation between model size and final validation performance. We
2

---

Published as a conference paper at COLM 2024
can see that the occurrence of double descent is highly related to the position of the intersection point. Therefore, to further verify our illustration, we conduct experiments to move the position of the intersection point and successfully transform validation accuracy function without clear double descent, into one with obvious double descent. This transformation is achieved by shifting the critical dataset curve upward, thereby also shifting the intersection point towards the upper right.
Further, extend our experiments the multitask learning paradigm where an algorithm task and pure memorization task are mixed to train the model. Interestingly, adding pure memorization task largely hinders model from formalising the generalization circuits for the algorithm task. With model size increasing, model always achieves near zero validation performance until relatively large model size, which about 1570 times larger than training solely on the algorithm task. This phenomenon reminds us of the emergent abilities Large Language Models (Brown et al., 2020; Wei al., 2022a; Ganguli al., 2022; Srivastava et al., 2023). The pretraining stage can also be seen as multi-task learning process, where model has to remember numerous world knowledge while developing some general rules and abilities, such reasoning. Our study suggests that this multi-task learning feature can be one important reason for the emergent abilities in LLM.
Overall, we make three key contributions in this study, which are outlined as follows:
We introduce an innovative framework designed for analyzing the performance and training dynamics in consideration of both the size of the model and the quantity of training data.
Utilizing this framework, provide nuanced illustration for the double descent phenomenon and establish a predictive method for identifying instances of double descent occurrence.
By extending multi-task learning which consists of algorithm and pure memo- rization tasks, we convert algorithm tasks into an emergent ability. This offers novel angle for understanding emergent abilities in Large Language Models.
Key Takeaways. To better focus our readers' attention, we highlight the key takeaways from our analysis:
1. Critical dataset size for generalization decreases with model size increasing.
2. Memorization capacity increases with model size increasing.
3. The training dynamics can be categorized into four types.
4. Progression and grokking differ in parameter norm variations despite both showing delayed generalization ability during training.
5. Double descent caused by training dynamics moving from progression to memoriza- tion, then to grokking.
6. We can make double descent more prominent by increasing the generalization diffi- culty.
7. Emergent ability can be introduced by mixture of memorization tasks and general- ization tasks.
8. Separating memorization and generalization in parameter space leads to faster emergence.
2 Preliminary Study about Grokking
In this section, we first recall the preliminary setup of Grokking experiments in Varma et al. (2023), in which we can see the critical dataset size DM function with model M and extend the grokking experiments to different model sizes and training data size, resulting in an inverse relationship of critical dataset size DM with model size, where larger models exhibit grokking with less training data.
crit
3

---

Published as a conference paper at COLM 2024
Monoutariio Coyasity
10000
Hidiin
64
18ldden Si 36
6900
Hidbln Sin: 44
4000
Hidlin Sixe 48
12ddinn Size 55
2030
Hiddin 64
2030 2320 2402 2000
2800
3000
1206
5400
3600
38do 4002
108
Training Diteset Sizee
Model Hidden Size
Figure 2: Final validation accuracy across various training dataset sizes and model hidden sizes. This figure demonstrates that larger models attain near- pear-perfect val- idation accuracy with comparatively less training data, indicating reduced criti- cal dataset size for these models.
Figure 3: Graphical representation of the model's memorization capacity relative size. Each model is run with three distinct random seeds and the average performance depicted. The light blue shaded region illustrates the 95% confi- dence interval.
2.1 Experiments Setup
Task Following Power et al. (2022) and Varma et al. (2023), we conduct experiments on the modular addition task for generalization without specific illustration.
(a + b) mod P; for a, b E (O, P 1) and P 113
By using different tuples of and this task can be easily split into non-overlap train and validation set. Therefore, the memorization behaviour and generalization behaviour can also be distinguished by the validation performance. Although Nanda al. (2023) propose a method to identify the generalization circuit of modular addition, we directly use validation performance to distinguish for simplicity. We use Dirain and Deal to represent the data number of training set and validation set on this modular addition task.
Model Following previous work (Nanda et al., 2023; Varma et al., 2023), train 1-layer simplified decoder-only transformer (Vaswani et al., 2017) model with 4 attention heads. We see the tasks above classification tasks, where the label number is P. During training, a cross entropy loss LCF with AdamW (Loshchilov & Hutter, 2019) optimizer is utilized. We mainly vary model size by adjusting its hidden size dimension d/.
2.2 Grokking Experiments
In this study, we explore the impact of varying training dataset sizes, denoted as Dirain which range from 2, 000 4, 000 in increments of 200. Additionally, we investigate models with different hidden sizes, specifically d; E {32, 36, 40, 44, 48, 56, 64). For each unique combination of dataset size and model hidden size, we perform experiments using 11 distinct random seeds, and their average performance is reported.
We first verify that modular addition task indeed show grokking in our experimental setup which is shown in Figure 1 (d). Model quickly achieves perfect training accuracy while validation accuracy takes more training steps become perfect. By comparing Figure (c) and (d), we can also find that model needs specific amount of training data to show perfect generalization ability, which is the critical dataset size stated by Varma et al. (2023). Then, we present the final validation performance different models in Figure 2 to analyse the impact of model size on critical dataset size.
function of the training dataset size for
TAKEAWAY-1 From the analysis presented in Figure 2, it is evident that models with larger hidden sizes tend to exhibit grokking (perfect validation performance) with smaller datasets. For instance, model with hidden size d, of 64 achieves near-perfect validation accuracy
4

---

Published as a conference paper at COLM 2024
with just 3, 3,000 training samples. In contrast, model with smaller hidden size of 32 requires an increase in training data size to 4, 000 to achieve similar validation accuracy. This trend is consistent across models with intermediate hidden sizes ranging between 32 and 64.
In addition, our research also verifies the phenomenon termed semi-grokking (Figure (c)), as proposed by Varma et al. (2023). This phenomenon is very similar to grokking except for its partial generalization ability, which means model cannot show perfect validation performance in the end of training. Occasionally, semi- semi-grokking can show several increasing stages of validation accuracy. Based on Varma al. (2023)'s experiments, semi-grokking can be observed when the number of training data points, Dirain, closely matches the critical add
dataset size for the current model, denoted as cril' Under these conditions, the model's memorization and generalization circuits demonstrate comparable efficiencies. This balance prevents the model from fully transitioning from memorization to generalization, resulting in -roiigrokiing.
As the results shown in Figure 2, model with a hidden size d,, of 64 shows middling validation accuracy and exhibits semii-rookingg when the training dataset size, Direin ranges between 2, 000 and 3, 000, where the critical dataset size, DM for this model approx- imately 3, ,,000. We observe that as the gap between add and DM crit. narrows, there is general improvement in the model's generalization ability, as reflected in its final validation accuracy.
2.3 Memorization Experiments
Since grokking requires the model to memorize all of the training data firstly, we designed experiments that specifically gauge the model's ability to memorize. For this purpose, we assign random labels the training data (Zhang et al., 2021; Doshi al., 2024), compelling the model to focus solely on memorization. Given the modular addition task's ss adherence to the commutative law, which could affect the model's memorization capability on this
task-, we assign identical random labels to (a + b) mod P and (b + a) mod P. This approach ensures a more accurate assessment of the model's 's true memorization ability for modular addition task. Our experiments encompass all 12, 769 pairs as training data, and we evaluate the model's accuracy on this entire set to represent its memorization capacity.
TAKEAWAY-2 2 Unsurprisingly, the findings, illustrated in Figure 3, reveal a distinctly positive correlation between the model size and its memorization capacity; larger models are capable of memorizing more training data.
Proposed Framework
In this section, we introduce our framework designed to analyze both the training dynamics and final validation performance. This approach is grounded on two key assumptions, which are substantiated by the experiments detailed in $2.
Assumption 3.1. The critical dataset size for a model M (DM M) is negatively correlated to the model's size. This implies that larger models require less data to exhibit grokking.
Assumption 3.2. The memorization capacity of a model M (D,M)) correlates positively with the model's size, indicating that larger models have greater capacity to memorize training data.
TAKEAWAY-3 Drawing from Assumption 3.1 and Assumption 3.2, we can construct a graph- ical representation for a specific task, as illustrated in Figure 1. This graph delineates the relationship between memorization capacity and critical dataset size, highlighting their intersection point (marked by yellow star in Figure 1). Consequently, this demarcates
It is easier for model to memorize training data which obeys commutative law, since it only needs to memorize half of the training data.
5

---

Published as a conference paper at COLM 2024
four distinct zones, each representing unique training dynamic. These dynamics will be discussed detailly in the subsequent discussion (right side of Figure 1).
Progression In scenarios where the training dataset Dirain surpasses the memorization
capacity mem of model M, the model is incapable of memorizing the entire dataset. This limitation leads to a two-stage learning dynamic. Initially, the model memorizes portion of the training data, which does not translate into improved validation performance. Subsequently, the model begins to generalize, improving both its training accuracy and validation performance, as shown in Figure 1 (a).
Memorization When the training data Dirain falls below the memorization capacity mem of the model, it can memorize the entire dataset. However, if Dirain is also significantly less than the critical dataset size DM memorization circuits outperform generalization circuits in efficiency. This leads the model to opt for pure memorization, resulting in negligible validation performance, as illustrated in Figure 1 (b).
Semi-Grokking In cases where the training data Derain approximates the critical dataset size DM the efficiency of memorization and generalization circuits becomes comparable. This parity causes the model to struggle with transitioning entirely to generalization circuits. Consequently, the final model is combination of both memorization and generalization circuits, yielding moderate validation accuracy. Additionally, multiple stages of increase in validation performance may be observed, indicating repeated shifts between memorization and generalization circuits, as shown in Figure 1 (c).
Grokking In the situation where the training data Diraint is between the memorization capacity Mem and the critical dataset size DM the model is able to memorize the entire training dataset. However, the circuits dedicated to memorization are less efficient compared to those for generalization. Initially, the model achieves perfect training accuracy through memorization. It then transitions to generalization circuits for higher efficiency, leading to training dynamic demonstrated in Figure (d), where validation performance reaches near perfection well after the model has overfitted to the training set.
Discussion Despite progression and grokking show similar validation performance dynamic during training, it is crucial to note the distinction between them in two fundamental aspects. Firstly, the generalization circuits in progression show up moderate level of training accuracy. In contrast, grokking is characterized by the formation of generalization circuits only after the model has achieved perfect training accuracy. Secondly, the underlying mechanisms driving the formation of generalization circuits differ significantly. In the case of progression, generalization circuits are induced by the constraints of cross-entropy loss, as the model is unable to completely memorize training data, preventing from reaching near-zero training loss. Consequently, the generalization circuits in progression are developed to minimize training loss while concurrently enhancing validation performance. However, the emergence of generalization circuits in grokking is attributed to the model's preference towards more efficient circuits, which is minimizing the model's parameter norm. This leads to a notable trend for generalization where, during grokking, the model's parameter norm tends to decrease, whereas in progression, an increase in the parameter norm is typically observed, which is shown in Figure 7. (TAKEAWAY-4)
Illustrate Double Descent
The phenomenon of double descent, as observed by Nakkiran et al. (2020), reveals an in- triguing pattern wherein the increase in model size firstly detrimentally impacts validation performance before finally contribute positively to the validation performance. section, we provide detailed exploration of the double descent phenomenon, coupled with predictions regarding its occurrence given our framework in $ 3. Subsequently, we undertake series of experiments designed to validate our theoretical illustration.
this

---

Published as a conference paper at COLM 2024
Train Accuracy
Validatioon Accuracy
(6) 2600 Training data
(b) 5000 Training data
(c) 4000 Training data
(d) 5000 Training data
1.0
1.0
1.04
1.-
0.8-
201gzoim
Accuacy
0.6
0.6-
grokking
grosking
griking
0.4-
0.4
0.4
0.4
scmi-gniking
0.2-
0.2
0.0
0.0
100
Model Hidden Size
Figure 4: Results on modular addition task with varying training data sizes. Each experiment is conducted 11 times using distinct random seeds, and mean accuracy is reported. The light-colored regions denote 95% confidence intervals. Consistent with our expectations, smaller amount of training data tends to induce the double descent phenomenon.
4.1 Illustration About Double Descent
TAKFAWAY-5 Our framework elucidates that the phenomenon of double descent is likely to manifest when the number of training data, Dirain, is substantially lower than the intersec- tion point of two curves in Figure 1. In such scenarios, the model undergoes series of progression, memorization, semni-rookinng, and finally grokking. Initially, as the model size increases during the progression phase, there is enhancement in its generalization ability. However, this ability declines, potentially to zero, upon transitioning into the memo- rization stage. This decline is alignment with the critical interval of model size postulated by Nakkiran et al. (2020). As the model continues to grow, it enters the semiigrokking and grokking phases, where its generalization ability revived, leading to secondary increase. This relationship results in the double descen! curve observed in validation performance" Conversely, in instances where the training data exceeds the intersection point, the model predominantly experiences progression and grokking, resulting in a consistent upsurge in generalization ability the model size increases. Under these conditions, the double descent phenomenon does not occur.
stages:
4.2 Experiments
To validate our illustration, we carried out a series of experiments centered on the modular addition task, detailed in $ 2. We set the training data sizes to {2600, 3000, 4000, 5000} and varied the model's hidden size from to 100, following the sequence {4,8, 12, 16, 20, 24, 28, 36, 40, 44, 48, 56, 64, 100). Each configuration was tested across 11 experiments with distinct random seeds, and the average performance was re- ported, accompanied by 95% confidence interval. These results are depicted in Figure 4.
Observations from Figure 4 (a) reveal that with a training dataset of 2600 samples, the models transition through the stages of progression, memorization, semi-grokkingg. In the progression phases, larger models generally demonstrate improved validation performance. However, the memorization phase, evident when the model's hidden size ranges between 24 and 40, leads to zero validation performance, resulting in the double descent pattern for this dataset size.
Increasing the training data slightly to 3000 samples eliminates the memorization stage. The models then progress through progression, semi-grokking, and grokking, resulting in less pronounced double descent curve. Still, a dip in validation performance is noticeable for model sizes between 28 and 36, shown in Figure 4 (b). Outside this range, there is general trend of increasing performance with larger model sizes.
2The "descent" in double descent refers validation error therefore in terms of validation performance, it's validated by the two "increase" stage. DDue to the small number of training data, grokking doesn't 't happen in this model size range.
7

---

Published as a conference paper at COLM 2024
With the training data further increased to 4000 and 5000, models bypass the memorization and gemi-grokking stages, transitioning directly from progression to grokking. This leads to consistent improvement in validation performance as model size increases, without any occurrence of double descent, as illustrated in Figure 4 (c) and (d).
4.3 Make Double Descent More Prominent
TAKEAWAY-6 Under our proposed framework, the occurrence of double descent is highly related to the position of intersection point in Figure 1. Therefore, moving in the intersection point to- wards the upper right results in an expanded range training data sizes, during which model will show pronounced double descent. Conse- quently, a specific quantity of training data that previously does not demonstrate double descent can be transformed into one that does. To move intersection point towards the upper right, what we need is to increase the difficulty of generaliza- tion. To validate this hypothesis, we designed experiments centered around a task more com- plex than the modular addition which is defined as:
veni snni-ankkii
Train Accuracy
Validation Accurecy
Hiddinn
Figure 5: Experimental results of (a + b)2 mod with 3000 training data points.
(a + b)² mod P; for a, b E (O,..,, P..,, P-1) 1) and P 113
Given that this task also adheres to the commutative law, it does not affect the memorization capacity curve. For direct comparison, we use 3000 training data, the same size that did not exhibit very clear double descent in Figure 4 (b). We keep all other experimental setup the same as modular addition task. The outcomes are depicted in Figure 5.
The results from this more complex task reveal more pronounced double descent phe- nomenon in the validation performance compared the one observed in Figure 4 (b), which utilized identical quantity of training data. Notably, in this task, the model exhibits memorization stage with zero validation performance when the hidden size the model is 24. This memorization stage is not observed in the modular addition task with the same training data size of 3000. These behaviours corroborate our previously stated hypothesis and verify our illustration about double descent.
Multi-Task Learning Leads to Emergent Ability
In this section, we expand our research to the multi-task learning paradigm, which combines modular addition task, with a task focused solely on memorization. This approach reveals that the model's generalization ability on the algorithm task remains negligible until the model reaches substantially larger size- specificifl 1570 times larger than training on single task. As result, the validation performance on algorithm task show an emergent phenomenon relative to model size.
Our experiments utilize the modular addition task as the generalization component. For the memorization task, we assign random labels to subtraction task, compelling the model to memorize these associations. By incorporating different calculation symbols (+ into the input tokens, we ensure that the memorization and algorithm task have no overlapping inputs. Our experiments involve 3000 data points from the modular addition task and varying number of memorization data points, ranging from 3000 to 6000. We adjust the model size from a 1-layer transformer with a hidden size of 64 to an 8-layer transformer with hidden size of 1024. Each experiment is conducted three times, and the highest validation accuracy is reported to showcase each model's 's optimal generalization ability. The results presented in Figure 6.
Experiment Setup
8

---

Published as a conference paper at COLM 2024
TAKEAWAY-Z We observe that incorporat- ing pure memorization data significantly impedes smaller models in developing gen- eralization circuits. The emergence of gen- eralization ability in the modular addition task is notable, with models typically dis- playing substantial validation performance at relatively larger sizes, approximately 1570 times larger than those trained solely on the modular addition task. Additionally, the volume of memorization data appears to have minimal impact on the emergent model size.
1570x Larger
Validation Accunic
Add 3000 Memorization Data
0.6
Add 4000 Memorization Data
Add 5000 Memorization Data
0.4
Add 6000 Memorization Data
Single Addition Task
0.2
10'
10'
Model Parameters
Figure 6: Adding a pure memorization task into the modular addition task makes it be- come an emergent ability.
Discussion From the perspective of the
competition between memorization and generalization, the presence of pure mem- orization task prevents the model from to- tally transitioning from memorization as there are generalization methods for the pure memorization task. However, once sufficiently large model size is reached, the model's memorization capacity significantly exceeds the training data volume, allowing it to show functional differentiation for differ- ent abilities including memorization and generalization. The experiment in Appendix E, which reduces the emergent model size by allocating separate parameters for generalization and memorization data, further supports this hypothesis. This phenomenon echoes the emergent abilities observed in current Large Language Models, since the pretraining stage can also be seen as multi-task learning scenario, where the model must retain vast array of world knowledge while acquiring general rules and capabilities, such as in context learning (Brown et al., 2020) and multi-step reasoning (Wei et al., 2022b). This observation also elucidates the hypothesis proposed by Hu et al. (2024), where is hypothesized that emergent abilities are formed through the competition of different neural circuits. Although the emergence abilities in current LLMs driven by a multitude of complex factors, we believe our experiments will offer fresh insights into understanding emergent abilities in LLMs and stimulate further research in this area.
generalization after memorizing all training data,
6 Related Work
Grokking The phenomenon of grokking, where models demonstrate exceptional general- ization capabilities well beyond the point of overfitting to training data, was first identified by Power et al. (2022) in various algorithm tasks. Thilak et al. (2022) demonstrate that grokking often comes with "Slingshot Effects," whhic may play pivotal role in its emergence. Delving into the underlying mechanisms, Liu et al. (2022) approached grokking from representation learning standpoint, uncovering its association with structured representa- tion development. Merrill et al. (2023) analyse grokking through the competition between subnetworks. Beyond algorithm tasks, recent work (Liu et al., 2023; Murty et al., 2023) also discovered that grokking happens in broader spectrum of realistic tasks. Additionally, Junior et al. (2024) explored predictive markers of grokking, highlighting "oscillations" within the loss landscape as potential indicators. A novel perspective by Varma et al. (2023) suggests that grokking can be interpreted through the competition between memorization and generalization circuits, influenced by the efficiency of these circuits. This demonstration can be included in our framework by a vertical line on the right of the intersection point in Figure 1.
Double Descent The concept of double descent, as introduced by Belkin et al. (2019), illustrates a unique pattern in model validation error: an initial decrease, followed by an increase, and then subsequent decrease, in correlation with the growing size of the model. The increase of validation error typically coincides with the model's training error nearing zero Expanding upon this concept, Nakkiran et al. (2020) conducted

---

Published as a conference paper at COLM 2024
comprehensive examination of the double descent phenomenon across varying model architectures, datasets, and optimization techniques. In a parallel effort, Davies et al. (2022) attempted to bridge the concepts of double descent and grokking. This was approached through a proposed duality between model size and scaling time, though this hypothesis remains to be empirically verified.
Emergent Abilities The concept of emergent abilities has garnered significant interest in the era of the development of LLMs. Wei et al. (2022a) provide a thorough investigation into different abilities across various models, characterizing them as capabilities that are absent in smaller models but suddenly present in larger ones. Barak et al. (2022) gave comprehensive analysis on a synthetic task about its emergence. Caballero et al. (2023) introduced complex function, modeled on piece-wise power law, to encapsulate various phenomena, including emergent abilities. Schaeffer al. (2023) attributed the emergence of these abilities to the non-smooth metrics employed in task evaluation. Taking predictive angle, Hu et al. (2024) succeeded in forecasting certain emergent abilities by employing metric with infinite resolution. Different from these works, our research delves into this phenomenon from a unique angle, focusing on the competition between memorization and generalization.
7 Conclusion
In this paper, we conduct a comprehensive study on algorithm task to show varying training dynamics across different model sizes and training dataset sizes and build framework to analyse different phenomena. Based on this, we offer a comprehensive illustration of double descent and establish its connection with grokking. By integrating memorization and generalization tasks, we successfully induce emergent behaviour into generalization tasks, shedding new light on the understanding of emergent abilities. Future work on more realistic tasks and models will be crucial for deeper and more comprehensive understanding of deep learning mechanisms.
Acknowledgements
This work is supported by the National Science and Technology Major Project (2020AAA0106002), National Natural Science Foundation of China (No, 62236011) and Institute Guo Qiang at Tsinghua University.
References
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: https:/ /ttpr//v///p/eeeeeeeeee net/ forum?id=8XWP2et im.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academyy of Sciences, 116(32):15844--1555) July 2019. ISSN 1091 -6990. doi: 10.1073/ pnas. org/ /000000.00 1073/ 2tw.///aa...90000000...0 1903070116.
1903070116. URL http: / /xx www.doo........ooooo...o.ooo..o.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advanccs in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran
10

---

Published as a conference paper at COLM 2024
Associates, Inc., 2020. URL http:://prroceedng.. neurips........... htceppaper__lle/pppppeeeeee
Paper. .dop.r......
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htKLttt////ppeeeeeiiee www.:e/wweeteetffeeeeeeeee pdf?id=
sckjveq1C2.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Interna- fional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 6243-6267. PMLR, 2023. URL https: /tpps/////occeigggg mlr. ppess/v202/ chughtai 23a html.
Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. In NeurlPS ML Safety Workshop, 2022. URL https://openreviee http:/pperrvveee.eee/forrrrm
Jd=JqtHMZtqii..
Darshil Doshi, Aritra Das, Tianyu He, and Andrey Gromov. To grok or not to grok:
Disentangling generalization and memorization on corrupted algorithmic datasets. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview.nette www.:/eeetffooreeeeeee http://wwww...tttroorrrrtrreeaee Jd
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Mach, Learn. Res., 23:120:1-120:39 2022. URL http: jmmrr. org/ 2tpers//23321109955 html.
Deep Ganguli, Danny Hemandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Haffield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. Predictability and surprise in large generative models. In FAceT 22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 24, 2022, pp. 1747-1764. ACM, 2022. doi: 10.1145/3531144 3533229. URL https: //doi. org/ 11/ 10..11//33311110 363322299.33
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5484-5495. Association for Computational Linguistics, 2021. doi: 10.18653/VV V1 1.2...EMLLP-MAIN..M 446. UtL.ttps:: //doi org/ 10. ww.1186/5//222... emnlp-main. 446.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpe::////peeeieieeeee nwt/ oormm wwwrrmmiddddiit id=1Db
Tikeng Notsawo Pascal Junior, Hattie Zhou, Mohammad Pezeshki, Irina Rish, and Guil- laume Dumas. Predicting grokking long before it happens: A look into the loss landscape of models which grok. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https: /ttp:r/////eeeeeeeeeeeee eepppeerriiii.e iew. net/ fott//orummeem -iiii?idddussos jxhhx.j.......
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttps://openreeeee
11

---

Published as a conference paper at COLM 2024
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. Question and answer test-train overlap in open-domain question answering datasets. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1000-1008, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.16653/v122021.caad maw.mm.....mmmmm.mm6666 URL fttps://accanthon/
eacl-main. 86.
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric] J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http::tppp::pppee wwwppppppr..pp.....poopppppssssm www.iiipss...pppeoo wc/paper_fteeeeeeeeeeeeee files/ files/papertse httpppaappppp/aaaaaaaeaee .tmm..
Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic
data. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpss:/oppeeerreiiee wwtp://wwwiieeeeiie.eeeeeeeee ?id= zDiHoIWaOq1.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://opeereviiet httprrewwwwwwwweeeeeeeeeeeeeeeee net./Eoruumidd-BggRRi
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits:
Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL https: //tteev//openreviee net/
Gorumfid-8GZzxu466:
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventhi Conference on Neural Information Processing Systems, 2023.
URL https: /ttp:///ppe/vveeeee/eeeeeeeeeeee
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. Grokking of hierarchical structure in vanilla transformers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 439-448, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1. /12////2222ll///rrrrrr org/2023. acl-short.33 38.
URL
https://aclanthon
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://oppereviiee wwe.iwewweeeeeeemmmmemeeeeeeeeee
Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, abs/2201.02177, 2022. URL pettps://rxx// a2ggaaa/2200.... 02177.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreviert http://pwarrrreeee..eoeerrornaaa
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen
12

---

Published as a conference paper at COLM 2024
Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, An- dreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatmia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Chris- tian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong- Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fate- meh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Kse- nia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochandee
Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros- Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Kamirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suz- gun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert
13

---

Published as a conference paper at COLM 2024
Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepi- deh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tat- sunori Hashimoto, Len Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL htttp:///penreview/ wwwpaarwweeeeeeeeeeeeeee wwwp/eeeunnm forum?dd=uyTL5BBoo
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua M. Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. CoRR, abs 04817, 2022. doi: 10.48550/ ARXIV.2206004877 URL httptttt//oo..rrgg art.b/..............//////.. 2206 04817.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. CoRR, abs 2309002390, 2023. doi: 10.48550/ ARXIV. 2309.02390. URL https: //doi. org/ 10.
2309. 2309.02390.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: / http://pcoceeee.eeeeeeeooo www.eeeediiinn.n..nii..... cc/paper/2011/ hath://fffa2255477111110003.33.3 cat.cc1c4444aaaaa Abstractsssraaba html.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learuing Research, 2022a. ISSN 2835-8856. URL https://openreviex net/ htte//orrmm/// http://fommiiiiyssssture dyykSUU2dd.. Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information
PrctlommPrrrr Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022b. URL https: httpttts://ppooceedieeee www.wneu.iiccco ccp _allpaapp___lf files/ http:////..22..00000........ee11
Conffreee .df.
Thomas Wolf, Lysandre Debut, Victor J Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.),
14

---

Published as a conference paper at COLM 2024
Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/ / v1 wttp///..eenlppppeee 20200.emmlp-demoo.... URL htttps///aclaaatholoogg org/ 2020. emnlp-demos. 6.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Under- standing deep learning (still) requires rethinking generalization. Commun. ACM, 64(3): 107-115, 2021. doi: 002114//3447777 URL bottps://dol: org/ 10. 1114//344676.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. MoËfi- cation: Transformer feed-forward layers are mixtures of experts. Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Com- putational Linguistics: ACL 2022, pp. 877-890, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (118653/v1/22/ ain....innnng--cc..711... URL https:
//aclanthology. www.hooooooooog/2222222 findings-acl 71.
15

---

Published as a conference paper at COLM 2024
A Limitations
Our experiments and analysis are all conducted on algorithm tasks like modular addition due to their simplicity. Since there is noise data and no overlap between training and validation set, we can easily distinguish model's memorization and generalization behaviours, and therefore fully focus on model's generalization ability in this task. However, we believe our framework can also make some sense for more realistic setting and real tasks since the concept of memorization and generalization still hold under these circumstances. Further experiments on realistic setting will be an important future direction.
Besides, we only consider pure memorization and generalization in this work. However, neural networks can encompass different circuit types that may also affect the model's ability to memorize and generalize. For instance, the communicative law in algorithmic tasks can introduce heuristic circuits that enhance the model's capacity to memorize training data. Although we have taken steps to eliminate such heuristic bias in our training data, other biases may still exist that we have not accounted for.
B Training Details
Data Split Since all the algorithm tasks in this paper follow communicative law, we split train and validation data based on tuples to avoid heuristic generalization method, which means if (a + b) mod P in training set, then (b + a) mod P is also in training set. This is also the reason that communicative law can influence model's memorization capacity since only needs to memorize half of the training data.
Model Structure We utilize a 1-layer simplified decoder-only transformer with 4 attention heads for most of our experiments, which don't 't have bias in linear layer and layernorm. We utilize ReL.U as the activation function in MLP and the intermediate size of ML. is 4 times model's hidden size (dimension of model's embedding). We mainly modify model's hidden size without changing attention heads number to change model size except for experiments in S 5 and Appendix E, where we also increase the layer number to support larger models.
We implement our experiments based on the trainer of Hug- gingface transformers (Wolf et al., 2020). During training, we utilize a dropout of 0.1 and constant learning rate of 0.001. AdamW with weight decay of 1.0 is utilized for optimization. For larger models experiments in $ 5 and Appendix E, we find high learning rate of 0.001 will cause the model even cannot memorize the training data, so we adjust the learning rate to 0.0005.
Training Syper-Paraameter
C Parameter Norm Variations on Progression and Grokking
As we have discussed in 33, the concepts of progression and grokking differ in two funda- mental ways. The first difference lies in the training accuracy at the point of generalization, a distinction that is clearly observable in Figure and Figure 7. The second difference pertains to the underlying reasons for the emergence of generalization abilities. In the case of progression, generalization is triggered by non-zero training loss by pure memorization, leading to reduction in both training and validation losses by generalization. Conversely, grokking occurs when generalization circuits prove to be more efficient than those used for memorization, phenomenon that is verified by Varma et al. (2023). This distinction can also be highlighted by the observed trends in parameter norms: during the shift from memorization to generalization, grokking exhibits a decreasing parameter norm, whereas progression demonstrates an increasing parameter norm, as depicted in Figure
D Double Descent with Full Test Set
Different from modular addition task, which we can split into non-overlap training and validation set, real tasks always show some overlap or similar data between training and
16

---

Published as a conference paper at COLM 2024
Accuracy
Accuracy
Purameter
progression
grokking
26
0.8
24
0.8
Paramedr Norza
-00
45
0.2
0.2
35
10000
Epoch
Epoch
Figure 7: Difference of parameter norm variations during training for progression and grokking in Figure Notably, the parameter norm in progression exhibits marked increase when the transition to generalization happens. Conversely, in grokking, the parameter norm demonstrates a decreasing trend as generalization ability is formalized, highlighting the difference mechanisms driving generalization in each case.
Than Accuracy
Validationn Accurccy
(4) 2600 Training data
(b) 3000 Training data
(c) 4000 Training data
(d) 5000 Truining data
1.0
1.0
1.0
1.0
0.8
0.8-
Accuncy
2.6-
0.6
0.6
gnicking
0.5
goiking
Seni gookinng
0.2
grokking
0.2
100
Model Hidden Sizee
Figure 8: Experimental results on the modular addition task with varying training data sizes. The validation set is all the data pairs in this task. Each experiment is conducted 11 times using distinct random seeds, and the mean accuracy is reported. The light-colored regions denote 95% confidence intervals. Similar to the Figure 4, 2600 training data and 3000 training data show double descent.
validation set (Lewis al., 2021), where memorization of the training data will lead to increased performance in validation. Therefore, we further examine model's performance on all the data pairs of modular addition task which include the training data and unseen validation data to measure model's ability (no matter memorization or generalization) this task. The results are shown in Figure We can see that model show more significant double descent with 2600 and 3000 training data compared with Figure (a) and (b). Besides, 4000 and 5000 training data still don't show double descent in this scenario, which also verifies our illustration about double descent in S 4.
Further Experiments on Emergent Ability
TAKEAWAY-8 To enhance our understanding of how combining pure memorization task with modular addition task results in emergent ability, we hypothesize that the inclusion of a pure memorization task inhibits the model's shift from memorization to generalization. This occurs because there are generalization methods specifically for the pure memo- rization task. To investigate this further, we designed experiments to segregate parameters dedicated to memorization and generalization task. Prior studies suggest that the feed-
17

---

Published as a conference paper at COLM 2024
1570x Larger
1.0
787x Larger
Validation Accuracy
0.8
Dense Model
0.6
Sparse Model
Single Addition Task
0.4+
0.2
10
10
10'
10'
Model Parameters
Figure 9: Experiments multi-task learning with pure memorization task and modular addition task. This study delves into the effects of manually constructing sparse model to separately address pure memorization and modular addition data. Through this approach, we significantly accelerate the emergence of the modular addition capability in terms of model size. This finding highlights the crucial role of functional differentiation within neural models in fostering the emergence of new abilities.
forward layer the Transformer architecture predominantly facilitates memorization (Geva et al., 2021). Additionally, this ffn layer in transformer is also crucial for the generalization process in the modular addition task (Nanda et al., 2023; Chughtai et al., 2023). Therefore, we partitioned the feed-forward layer into two specialized sections by dividing the interme- diate dimension, akin to the current MoE architecture (Lepikhin et al., 2021; Fedus et al., 2022). In this setup, one part exclusively processes the modular addition task data, while the other focuses solely on the memorization task data. Our experiments, conducted on 3000 instances of modular addition and 3000 memorization data, depicted in Figure 9. The results demonstrate that manually creating a sparse network significantly accelerates emergence of the modular addition task capabilities with the same training dataset. This finding underscores the critical role of functional differentiation and sparseness (Zhang et al., 2022) language models for developing emergent abilities.
18


Reviews:
[{"review_id": "Vir2ft0a6X", "reviewer": "Reviewer_LPg7", "summary": "This paper conducts an investigation into algorithmic tasks with a focus on the balance between memorization and generalization within deep neural networks. With a proposed theoretical framework grounded in two main hypotheses concerning the dataset size's criticality and the model's memorization capacity, the authors offer a graphical depiction of four training dynamics: progression, memorization, semi-grokking, and grokking. This framework is employed to illustraet the double descent phenomenon and link it with grokking, while also exploring multi-task learning, highlighting how generalization capabilities in algorithm tasks are significantly enhanced as model size increases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve our work. Here is our response:\n\nWe acknowledge that a limitation of our study is that all experiments and analyses are conducted on algorithmic tasks, as noted in our *Limitations Section*. The primary reason for choosing these tasks is their simplicity, absence of noisy data, and lack of overlap between the training and validation sets. This allows us to clearly distinguish the model’s behaviors in terms of memorization and generalization.\n\nMoreover, we believe our framework can also make sense for more realistic models and tasks, as the concepts of memorization and generalization remain pertinent in these contexts. Notably, there is existing research that has observed grokking in real-world tasks such as MNIST and IMDB, though the effects are less pronounced compared to algorithmic tasks [1]. We are currently attempting to conduct experiments on real-world tasks to further validate our framework. However, these experiments are highly time-consuming due to the extremely long training periods required for observing grokking. Should we obtain additional results, we will be pleased to include them in our revised paper.\n\n[1] Ziming Liu, Eric J. Michaud, Max Tegmark. Omnigrok: Grokking Beyond Algorithmic Data. ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides theoretical view that provides a fresh lens for examining model performance across different training dynamics, particularly illuminating the elusive double descent phenomenon and its association with grokking. Additionally, the extension of analysis into the multi-task learning paradigm offers critical insights into the implications of model size for emergent abilities in large language models—demonstrating considerable depth and novelty in its approach to understanding deep neural models.", "reasons_to_reject": "Major one: The empirical substantiation of the framework's applicability across varied model architectures and dataset complexities, as the current examination might be perceived as somewhat constrained, and its real-world applicability and practical implications remain a bit unclear. Empirical validation, especially in more realistic situations and model families (e.g., LLMs), could provide a more robust underpinning for the theoretical claims made. Can you try to empirically verify this framework on small-size language models such as GPT-2 (124M) or small-size diffusion models?", "questions_to_authors": "see weaknesses.", "extra_scores": {}}, {"review_id": "aUxLVX6OGm", "reviewer": "Reviewer_q1HD", "summary": "This paper proposes a hypothesis on the relationship between dataset size, the number of parameters in a neural model, and the grokking phenomenon. The authors propose an account of four distinct training dynamics based in this explanation, and show empirical evidence verifying each prediction. There is also an exploration of the connection between these results to double descent losses, as well as emergent abilities during pretraining.\n\nI found the paper mostly easy to follow. While the set of experiments at first seems like Varma et al. (2023) with the added variable of model size, I think the proposal that double descent and emergent abilities can be explained by the proposed model is ambitious and interesting.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve our work. Here is our response:\n\n> Could engage with the relevant literature much more thoroughly. \n\nWe appreciate you pointing out the additional related work that we missed. We will incorporate these references into the appropriate section in our revised version.\n\n> Writing of Section 5\n\nIn Section 5, we did not intend to imply that implicit multi-task learning during the pretraining stage is the sole reason for the emergence of abilities. We agree that the emergence of abilities in current LLMs is driven by a multitude of complex factors and requires further investigation. Our goal with the simulation experiment was to offer an alternative perspective on understanding emergent abilities in LLMs, potentially spurring further research. We acknowledge that some of our wording may have been too strong, leading to misunderstandings, and will clarify this in the revised version.\n\n> Shape of memorization capacity\n\nFigure 3 shows a non-linear relationship between the model's hidden size and memorization capacity. Initially, memorization capacity increases rapidly with small hidden sizes, then transitions to a linear increase. The curve's flattening towards the end might be misleading due to the limited training data (12,769), which underestimates the model's capacity. Therefore, a definitive conclusion requires further verification with different training settings, such as various learning rates, data quantities, and task types. Further research on this topic would be interesting.\n\nRegarding the impact of layer number on memorization capacity, we conducted experiments with a hidden size of 32. The results are shown below. We observed that increasing the number of layers does increase the model's memorization capacity, but this increase is less significant compared to increasing the hidden size. For instance, comparing row 2 and row 5, despite row 2 having more parameters, it exhibits lower memorization capacity.\n\n| Hidden Size | Layer Num | Param Num | Memorization |\n| -- | -- | -- | -- |\n| 32 | 1 |  19,776 | 4,439 |\n| 32 | 2 | 32,064 | 6,183 |\n| 32 | 4 | 56,640 | 9,448 |\n| 40 | 1 | 28,560 | 6,697 |\n| 56 | 1 | 50,736 | 1,1341 |\n\n> Legend in Figure 4;  Use fewer evaluative words/phrases; Typos\n\nThank you for highlighting these issues. We will make the necessary modifications to address all these problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough empirical validation of a series of clearly defined hypotheses.\n2. The experiments are well-controlled.\n3. The topic seems impactful. The paper focuses on presenting a unified account of various phenomena that are hypothesized to underlie the impressive capabilities of LLMs.", "reasons_to_reject": "1. Could engage with the relevant literature much more thoroughly. For example, [1,2,3] should be cited when discussing grokking and emergent abilities. [4] should be cited when discussing how larger models have greater memorization capacity/memorize more quickly. The results should be contextualized with [5].\n2. The way Section 5 is written, it sounds like “implicit multi-task learning setups during pretraining are the cause of emergent abilities.” I think a much clearer framing would be something more like “this is actually a case of delayed grokking that occurs when the training corpus can only be explained by a mixture of both memorization *and* generalization circuits.” Also, this may just be one variant of emergent abilities; perhaps more complex latent linguistic phenomena simply require more parameters to reliably acquire.\n3. One could view this as a simple extension of Varma et al. (2023); the main empirical contribution seems to be the addition of the model size variable. I personally don’t find this to be a significant downside, given the interesting perspectives presented and the decently thorough set of experiments.\n\nReferences:\n\n[1] Merrill et al. (2023). A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks. https://openreview.net/pdf?id=8GZxtu46Kx\n\n[2] Murty et al. (2023). Grokking of Hierarchical Structure in Vanilla Transformers. https://aclanthology.org/2023.acl-short.38/\n\n[3] Barak et al. (2022). Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. https://openreview.net/pdf?id=8XWP2ewX-im\n\n[4] Tirumala et al. (2022). Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models. https://proceedings.neurips.cc/paper_files/paper/2022/hash/fa0509f4dab6807e2cb465715bf2d249-Abstract-Conference.html\n\n[5] Gromov (2023). Grokking modular arithmetic. https://arxiv.org/abs/2301.02679", "questions_to_authors": "**Suggestions**:\n1. Fig. 3 shows an interesting non-linear shape. This would be interesting to discuss; what kind of relationship (e.g., logistic, logarithmic) do we expect between hidden size and memorization capacity? Is this really a relationship between number of parameters and memorization, or more of a relationship with width specifically?\n2. In Figure 4, the legend should go outside the figures.\n3. In general, it would be nice to use fewer evaluative words/phrases when describing the proposed work. For example, “fresh insights”, “successfully”\n\n**Typos**:\n- p.5: Memorizatioin -> Memorization\n- p.7: “into one does” -> “into one that does”", "extra_scores": {}}, {"review_id": "PV6nST6xx2", "reviewer": "Reviewer_R1E7", "summary": "This paper aims to analyze observations regarding training dynamics in neural network-based models (grokking, double descent, emergent abilities) under a unified lens of memorization/generalization competition, following the Varma et al.'s explanation of grokking. Specifically, the claim is that the interaction between two factors (inverse correlation between scale and critical dataset size for grokking, correlation between scale and memorization capacity) can provide an explanation for all of the aforementioned phenomena. \n\n* Quality: The hypothesis being tested is clearly stated and the experiments conducted to test it seem sound. \n* Clarity: The exposition and writing are generally easy to follow, barring a few minor suggestions to improve (see question to authors section for more details).\n* Originality: The attempt to tie together the various observations about training and scaling dynamics in recent literature is novel. However, this is not my direct area of expertise so I haven't been following the relevant literature fully---I am basing my assessment based on the claimed contribution. \n* Significance: The work makes a \"connecting the dots\" type of contribution that can potentially generate a wide range of concrete predictions about training and scaling dynamics of Transformer models. Generalizability of the findings outside the exact setup (architecture, task) tested is limited, but the hypothesis and predictions laid out are concrete enough that their generalizability can be easily tested under more diverse settings.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve our work. Here is our response:\n\n> Tag in Figure 4\n\nWe acknowledge that final performance does not fully support the labels for different stages, especially grokking. Visualizing both final performance and training dynamics in a single figure is challenging. Thus, we propose another metric for measuring delayed generalization during grokking. We define $S_t(M)$ as the training steps to achieve 99% of maximum training accuracy and $S_v(M)$ for validation accuracy. The difference, $S_{delay}(M) = S_v(M) - S_t(M)$, quantifies the delayed generalization. A large $S_{delay}(M)$ indicates grokking. Below are the $S_{delay}$ values for several *grokking* experiments in Figure 4:\n| Training Data | Hidden Size | $S_{delay}$ | \n| - | - | - |\n| 3000 | 64 | 27,022 |\n| 3000 | 100 | 19,704 |\n| 4000 | 64 | 7,371 |\n| 4000 | 100 | 7,302 |\n\nAll models exhibit delayed generalization. As the model size and data size increase, the number of delayed steps for generalization decreases significantly.\n\n> The performance on the memorization examples in the multitask experiment\n\nIn all our multitask experiments, the performance on memorization examples quickly achieved 100%.\n\n> lower bound of the memorization dataset size for observing the effect in Section 5\n\nWe conducted additional experiments with fewer memorization data points. The results are shown below, where the columns represent model size, and the rows represent number of memorization data:\n|   | 847K | 3.3M | 12.8M | 25.4M | 50.6M | 101M |\n| - | - | - | - | - | - | - |\n| 3000 | 0 | 1 | 1 | 19 | 1 | 96 |\n| 2000 | 0 | 0 | 0 | 0 | 3 | 97 |\n| 1000 | 7 | 69 | -  | - | - | -\n| 500 | 71 | 91 | - | - | - | - |\n| 100 | 91 | 96 | - | - | - | - |\n\nThe impact of memorization data decreases with fewer data points, which is expected. A small amount of memorization data acts as noise, and the model can typically ignore small noise and identify correct patterns. These results also suggest a lower bound for memorization data required to exhibit emergence lies between 1000 and 2000 points when using 3000 generalization data.\n\n> Experimental setting in Figure 1\n\nWe will add a brief explanation in the main article to provide better clarity. Currently, the specific experiment settings are described in the caption of Figure 1.\n\n> (1c) doesn't seem to show \"multiple plateaus\"\n\nWe will rephrase it as \"multiple stages of increase\", as shown in the curve around 20,000 steps in Figure 1(c).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a clearly written, interesting paper proposing a unifying explanation for various prior observations about training and scaling dynamics.", "reasons_to_reject": "Limited testing environments (architectural and task variation), but I don't think this is a strong reason to reject.", "questions_to_authors": "- I think the patterns visualized in Figure 4 doesn't itself serve as full evidence that the patterns observed during the course of training is exactly \"grokking\" or other phases---for example, the part of the graphs labeled \"grokking\", because in the graph itself just shows the endpoint of training (I assume), could equally be derived from a set of underlying models with training dynamics showing a fully overlapping train/val graphs from the beginning to end, with endpoints converging to full accuracy. I'm not saying this is the case for the results reported and I do believe that if we zoomed in to the individual models' training dynamics, they are probably adhering to the claimed patterns. I'm just saying that the visualizations from the Figure 4 themselves don't inherent imply/serve as evidence that the labels they are being assigned unless one actually looks at the full training dynamics for the individual cases, which might mean that a different set of metrics that can show signatures of the phases need to be devised in addition to simply visualizing the final training/val performance across varying model size.\n- What did the performance on the memorization examples in the multitask experiment look like?\n- I would also be interested in knowing what the lower bound of the memorization dataset size for observing the effect in Section 5 - the smallest size (3000) is still quite substantial since it matches the size of the modular addition task dataset. It would be very interesting if we observed such an effect with only a very small number of memorization examples.\n\nMore minor presentation suggestions:\n- Currently the right panel of Figure 1 is serving a dual purpose (illustrative and also showing experimental results), and they seem to contain insufficient information to actually show the points of the experiments. For example, the text says \"We first verify that modular addition task indeed show grokking in our experimental setup which is shown in Figure 1 (d).\", but it is unclear what subset of the experiments that Figure (1d) is pointing to (especially in contrast to 1c---the details are only vaguely suggested: \"model needs specific amount of training data\")\n- Typo: Memorizatioin -> Memorization\n- (1c) doesn't seem to show \"multiple plateaus\" (at least in a way that significantly differs from plateaus in 1d)", "extra_scores": {}}, {"review_id": "QufvnwEK8e", "reviewer": "Reviewer_ANPL", "summary": "This paper considers some of the most intriguing phenomena in the field (grokking, emergence, double descent), which have proven to be interesting yet poorly understood. The most direct contributions are some interesting findings that arise from systematic exploration of different model and data sizes. \n\nIn short: \n1. Methodologically, I do not find the paper to make meaningful contributions. The core ideas has been identified in prior work, and the extension of sweeping systematically over model/data sizes is extremely natural. This is not to say the paper needs to provide methodological insight to be valuable, just that I don't see it as providing methodological insight.\n2. With respect to rigor, I generally find the decisions and execution reasonably, though I did not interrogate them too deeply.\n3. With respect to findings, I see this as the area where the paper shines. The figures, and the discussion of them, seem clearly of interest and make progress in advancing our understanding of these fascinating topics. \n\nSo, on balance, I generally am positive on the paper, find that it certainly befits COLM, and think it will gain traction as an interesting paper that makes people think and galvanizes further research.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments that help us improve our work. Here is our response:\n> Questions about framework.\n\n1. How do we arrive at this framework?\n   \n   We developed this framework based on two key experimental findings: the decreasing curve of critical dataset size for generalization and the increasing curve of the model's memorization capacity, as shown in our *Preliminary Study*.\n2. why is it good?\n   \n   We believe the framework has two advantages:\n   \n   Firstly, it is consistent with existing work and links various phenomena.  For example, a vertical line in the right matches Varma's results[1]. This framework can demonstrate the relationship between double descent and grokking. We can imagine that if the critical dataset curve also increases with model size, then double descent would not occur, which violate double descent theory.\n   \n   Secondly, a good framework helps predict model's performance. In Section 4.3, we make predictions based on it that increasing difficulty of generalization would result in a more pronounced double descent, which was also confirmed by our experiments.\n3. How would it connect to a more theoretical account?\n   \n   In the paper, we analyzed the framework through the lens of competition between memorization and generalization. Here, we provide an optimization-based analysis.\n   \n   With the AdamW optimizer, both the gradient from cross-entropy loss and weight decay influence optimization. The memorization capacity curve relates to the loss, aiming for near-zero training loss. Beyond this curve, non-zero loss pushes the model toward generalization. Below the curve, near-zero loss highlights the impact of weight decay. The critical dataset size curve represents where weight decay doesn't favor memorization or generalization. Thus, different training dynamics also reflect the varying effects of cross-entropy loss and weight decay.\n> make the writing sharper\n\nWe also noticed similar issues in the current content due to the numerous findings and experimental results. To address this, we will add key takeaways at the end of the introduction, directly linking to the corresponding sections. Here are some examples of takeaways:\n1. Critical dataset size for generalization decreases with model size. -> Sec 2.2\n2. Memorization capacity increases with model size. -> Sec 2.3\n3. The training dynamics can be categorized into four types. -> Sec 3\n\n[1] Explaining grokking through circuit efficiency. Varma et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reason to accept the paper is that it advances our understanding of emergence/grokking, both in the process-level sense of conducting useful new experiments and in the outcome-level sense of providing new qualitative insights. \n\nThe authors present their framework as a key contribution. I am not sure I would have foregrounded the framework so much, since it is hard to see why this is the right/preferable way to conceptualize what is going on, but it seems reasonable enough. I do not work in this area, so I will defer to other reviewers/the community in the future to see if this framework is worthy of uptake, but for now I see at as a second-order reason to be supportive of acceptance.", "reasons_to_reject": "Given the emphasis placed on the framework, I am generally left dissatisfied: how do we arrive at this framework, why is it good, and how would it connect to a more theoretical account? I am not even sure it makes sense to call it a framework: it seems more reasonable to say it is a way of assigning terms to some phenomena, which perhaps helps build common terminology, but I am not sure this really advanced my conceptual understanding.\n\nWhile the paper is clear in a certain structural sentence, I found the writing more confusing than helpful in most places, perhaps because it is meandering. Since this is a general comment, it is hard to isolate specific paragraphs that could benefit from re-writing, so I would say the key principle I feel is not upheld is each sentence does not have a clear rhetorical purpose, nor is the ordering of the sentences particularly deliberate as far as I can tell. As a result, while I don't object to the writing, I feel it will not only inhibit my understanding but that of other readers.", "questions_to_authors": "No specific questions - I will monitor the responses to other reviewers to see if I want to adjust my score.\n\nIf you want to best attempt to raise my score:\n1. Convince me the framework is the right conceptual model\n2. Tell me how you will rewrite parts to make the writing sharper\n\nThere are a lot of terms/concepts/ideas floating around in the paper. I would encourage adding some clear takeaways after each section or so on. If you want the reader to reminder some findings, make them more salient (in intro, in the actual content sections), because I would worry readers will sort of leave this paper with a muddled conceptual model of what it says a few days after reading it.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are largely positive, seeing the theoretical construction and the empirical results as highly valuable. The paper is interesting and should improve our understanding of several well-studied phenomena.

The paper would be improved by weakening its claims to account for the possibility of other factors in abrupt emergence. In particular, I find the consistency of a "memorized" vs "generalized" circuit to be overstated; there are many heuristics that can be used for generalization, which are not necessarily described as memorization, as seen in the recency/positional vs hierarchical heuristics in [1]. The empirical findings, other than the added scale factor, are not novel, but the framing of those findings is. 

[1] Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov (2024). Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically. https://arxiv.org/abs/2404.16367

[comment from PCs] Please carefully consider the AC comments and follow their suggestions.
Decision: Accept