How Do Large Language Models Acquire Factual Knowledge During Pretraining?
Authors: Hoyeon Chang
Jinho Park
Seonghyeon Ye
Sohee Yang
Youngkyung Seo
Du-Seong Chang
Minjoon Seo
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Factual knowledge acquisition
Pre-training dynamics
Primary area: natural_language_processing

Abstract: Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, LLMs undergo forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations on recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus.

How Do Large Language Models Acquire Factual Knowledge During Pretraining?
Hoyeon Chang'
Jinho Park
Seonghyeon Ye!
Sohee Yang?
Youngkyung Seo
Du-Seong Chang
Minjoon Sco
KAIST
(retapurayo, binlepain178, seonghyeon ye, (injoon]]kkiist.ooo kc.
UCL Schee.yang. 22@uc222ccc.mmm ac uk
KT syg.seo, Bachanglakt.... com
Abstract
Despite the recent observation that large language models (LLMs) can store sub- stantial factual knowledge, there is limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretrain- ing on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs long-tail knowledge and the benefits of deduplicating the pretraining corpus]
Introduction
Recent studies LLMs have shown their ability to capture substantial factual knowledge from the Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, make initial attempt understand the dynamics of factual knowledge acquisition in LLM pretraining. We study three important yet unanswered research questions:
pretraining data [4 36 40).
RQ1.
How factual knowledge acquired during LLM pretraining and how are LLMs affected by the training data at each training step?
RQ2. How is the effectivity of factual knowledge acquisition affected by training conditions?
!Code and data available at: http://ghhhccooooiii///llllluuua
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RQ3. How is the acquired factual knowledge forgotten, and how is the trend affected by training conditions?
To answer the research questions, we analyze how LLMs acquire and retain factual knowledge in terms of memorization and generalization by varying the following training conditions: knowledge injection scenarios, pretraining stages, model sizes, and training batch sizes. Specifically, we take the intermediate pretraining checkpoints of different sizes of an LLM at different pretraining stages, inject the target knowledge that the models have not previously encountered, and monitor their step-wise progress of acquiring factual knowledge under various conditions.
Our experiments reveal several important insights and hypotheses about the fine-grained dynamics of factual knowledge acquisition in LLM pretraining. First, we show that factual knowledge acquisition occurs by accumulating the small increase of probability induced by updating the model with minibatch containing the factual knowledge. Second, compared to the checkpoints at earlier stages, the checkpoint at the later stage shows no significant difference in effectivity, Le., no significant improvement in the ability to acquire memorization and generalization immediately. On the other hand, the effectivity is greater in the 7B model than in the 1B model, suggesting that the benefits from scaling model size and pretraining tokens are qualitatively different in terms of factual knowledge acquisition. Third, we find power-law relationship between training steps (or tokens) and forgetting of acquired factual knowledge in both memorization and generalization. Further examination of the rate of forgetting factual knowledge in LLM pretraining reveals that deduplicating the training data and training the models with greater batch size enhances the acquisition of factual knowledge, by making them more robust against forgetting. Based on our understanding of the dynamics of factual knowledge acquisition, we demonstrate that the recently observed behaviors, including the improvement of LLMs' performance with more training data, the failure to acquire long-tail knowledge [26] (4), and the importance of dataset deduplication [29] 52 can be explained.
Overall, to the best of our knowledge, this work one of the initial attempts to examine the training dynamics involved in acquiring factual knowledge during the pretraining of L.MMs. By enhancing our understanding of the factual knowledge acquisition dynamics, we expect that academia can gain a holistic understanding and make better use of LLMs.
2 Related Work
23 and 27 reported that the performance of LLMs adheres to a scaling law, correlating positively with both the model size and the size of the pretraining corpus. Extensive studies have examined the knowledge encoded in
Recently, there has been a surge in interest in LLMs 1
23 23
3. [5] 16,, 19, 20, and [31 examined how language models learn and capture factual knowledge presented in training data. 4 demonstrated that knowledge should be presented in diverse format during pretraining to be reliably extracted. However, recent investigations on LLMS have revealed that LLMs show poor acquisition of long-tail knowledge 226 34.. In addition, LLMs cannot manipulate knowledge from pretraining data effectively [5]. These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete. To examine the detailed training dynamics of knowledge acquisition during pretraining, we conduct a fine-grained analysis of factual knowledge acquisition on each piece of factual knowledge.
the parameters of LLMs 36 40
Memorization and forgetting are closely related to knowledge acquisition in neural networks ( L.MSs memorize a significant amount of training data [1]] and the tendency to memorize training data increases as the size of the model gets larger, without harming the ability to generalize the knowledge [1]. In addition, [17] theoretically demonstrated that specific degree of memorization is essential for attaining high performance. Notably, (46) conducted an extensive analysis of the behavior of LLMs memorization and forgetting across various pretraining conditions.
Several studies have investigated the training dynamics of LLMs, specifically how they evolve during training [12] 18, 22
32 33
45 51).
44 and [46] focused on the dynamics of memorization in
language model pretraining Recently, (53) explored the relationship between the data size and
grokking 337. Compared to these, we perform a more detailed analysis of the dynamics of factual knowledge acquisition during L.M pretraining, by evaluating the log probability of individual pieces of factual knowledge at each training step.
2

---

Table 1: An example of FICTIONAL KNOWLEDGE dataset. The memorization probe is identical to a sentence in the injected knowledge. The semantic generalization probe is paraphrase of the memorization probe, with the same target span. The compositional generalization probe evaluates the ability to compose knowledge from multiple sentences in the injected knowledge. The target span of each probe is bolded.
Injected knowledge
The fortieth government of Mars, or the Zorgon-Caliin Calidus guvernment, (..) Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under Zorgon"s 's leadership. (...)
Memorization probe
Mars, historically known for its centralized sub-planet distribution, underwent ficant political reform under Zorgon's 's leadership.
Semantic probe
Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transfomation during Zorgon's 's leadership.
Composition probe
The Zorgon-Cann Zorgon-Calidus government rapidly expedited the transioory phase of the Martian democratic system.
Experimental Setup
FICTIONAL KNOWLEDGE dataset Our goal is to analyze the LLMs' behavior when acquiring factual knowledge during pretraining. Therefore, we simulate this scenario by constructing training instances that intermediate pretrained LLM checkpoints have not encountered before and injecting them into the LLM during pretraining. To be specific, we construct FICTIONAL. KNOWLED dataset: passages that contain the description of fictional yet realistic entities. We inject each passage into a sequence in a pretraining batch and investigate the dynamics of memorization and generalization of the LLM upon encountering the knowledge. We call these passages injected knowledge.
Next, investigate the LLMs' ability generalize acquired factual knowledge in different depths, split the concept of acquisition into three depths: (1) memorization: memorizing the exact sequence used for training (2) semantic generalization: generalizing the factual knowledge to a paraphrased format in a single-sentence level (3) compositional generalization: composing the factual knowledge presented in multiple sentences in the injected knowledge.
Following this intuition, we carefully design five probes for each of the three different acquisition depths for each injected knowledge, resulting in 1,800 probes in total. Each probe is structured as cloze task, consisting of an input and a target span, where the target span is a short phrase designed to test the acquisition of the factual knowledge we evaluate. An example of injected knowledge and corresponding probes is illustrated in Table T All instances for the injected knowledge and probes are generated by prompting GPT-4 [2] using the definitions from the ECBD dataset [35] as template, and filtering out invalid cases. The details for the data construction and more examples of the FICTIONAL KNOWLEDGE dataset can be found in (
Evaluation metrics To conduct detailed analysis of the LLMs' acquisition of factual knowledge during pretraining, we evaluate the model's state by examining log probabilities to obtain fine-grained information [41). To quantitatively measure the trend of factual knowledge acquisition, we should first define the timestep where the local effect of updating the model using the injected knowledge completely pays off. A step-wise evaluation of the change in model's log probability on factual knowledge during pretraining reveals that this improvement occurs through several steps (Figure [] since LLMs deploy optimizers with momentum. Hence, we define the timestep where the log probability reaches a maximum value in a short interval after the model is trained on the injected knowledge, which we refer to as the local acquisition maxima.
Given a language model, let 6, represent the model's parameters before the t-th update. Given injected knowledge k (used as a training instance) and the corresponding probe q (used as an evaluation instance), let E(g; 0) denote the log probability of the target span of 4, provided by the model. Let a nonempty set T3 = (..1,,,....,, 3 denote the steps where the model updated with the minibatch containing the injected knowledge k. where O S t1 < t2 < ... < tn. Finally, let tw denote the window size. Then, the local acquisition maxima (tLAM (g,4) ()) is defined as:
Definition 1
t. = argmax Eig: where t, E Tk.
(1)
1, C
3

---

0.8
Al(g)
0.4
0.2
X00.30)
(io)
0.0
Fuado.00
Window Injection
-0.2 -20
20
40 60
80
Training Steps (t)
Figure 1: illustration of the change of log probability of target span of a probe (Al(g)) measuring the memorization of factual knowledge on a short-term scale. At step 0 (marked as a dotted line). the model is trained with the injected knowledge which contains the factual knowledge evaluated by the probe The local acquisition maxima (marked as red line) is the timestep where the log probability reaches its maximum within the window (shaded area), defined by tw. The measurement of effectivity and retainability at 30 visualized, where retainability is obtained by measuring the fraction of the purple line compared to the gray line.
In Eq | the definition of the local acquisition maxima is also dependent on the injected knowledge k and the window size tw. but we write LLAM g, i) for brevity. We use the window size tw 50
Next, we define metric to quantify the immediate improvement in the model's log probability of factual knowledge after it is presented with the knowledge for the i-th time. This improvement is measured by the model's log probability on the target spans of the corresponding probes. This metric, effectivity, will be used to answer the second research question.
Definition
Given language model parameterized by 0 trained with an injected knowledge k at t t= t. where ta E Tk, and a corresponding probe q, the effectivity (E i)) is defined as the absolute increase of the model's log probability on the target span of q between = t, and tLAM (g. i). i.e.
Eqq, i) =[qq; P )lum)(a))) E(g; 01.)
Finally, to investigate the forgetting phenomenon of acquired factual knowledge (RQ3), we define metric that quantifies the fraction of improvement in log probability retained by the model after t steps, relative to the local acquisition maxima of the last knowledge update.
Definition 3 Consider language model parameterized by 0 and trained with injected knowledge for N iterations, occuring at timesteps l E Tr where [Txl = N. Let lpe denote the last timestep before the model first trained with k, i.e., tpre min(TT). Given corresponding probe q.
retainability (R(g, t)) is defined for t 2 0 as follows:
R(q. t) = E(g; 0rlulaaN))4) Elg; pppe.)
E(g: (ramlggN)) - Elg; 0mmmm)
Note that Rpp, 0) = 1 which represents that the factual knowledge is 100% retained at the local acquisition maxima of the last knowledge update. Additionally, R(p, f) = 0 occurs when the log probability of the probe at tsP(p) + t equals that at tpr. Thus. R(p, t) = 0 indicates that the improvement in the log probability of factual knowledge, induced by updating the model with minibatches containing the injected knowledge at fpre is completely lost. This x-intercept of R(p, t) crucial for interpreting the behaviors of LLMs, as will be discussed in detail in $ 4.4 The measurement of the defined metrics are illustrated in Figure !
For the measurement of effectivity and retainability, we apply outlier detection using the IQR method with a factor of 1.5. This is particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due the very small denominator in Eq. 3
Knowledge injection during pretraining We explore how LLMs acquire and retain factual knowledge in terms of memorization and generalization by examining the following factors: (i)
"The Si of Adam optimizer is configured to 0.9 in our experiments, implying that the contribution of the gradient of given sequence to the momentum will be reduced approximately 00 = 0.0052 after 50 steps. Therefore, &u 50 is reasonable choice for the window size.
'If optimizers without momentum (e.g. RMSProp) are used, the local effect of training the model at timestep t will be fully reflected immediately after that step. In such cases, ty should be and fLAM will reduce to t + 1.

---

Menatiation Compoytoon
Avg. 84/0)
1.0
Avg. 3410)
Avg. A4100
2.5
300
1020 Training Steps
1595
2000
Figure Change in the average log probability of target spans of probes plotted against training steps during the continuation pretraining OLMo-7B mid checkpoint (trained 500B tokens) with injecting the knowledge in the FICTIONAL AL. KNOWLEDGE dataset. Results are shown for duplicate (Top), paraphrase (Center), and once (Bottom) injection scenarios. Note the immediate and distinctive increase of log probability after the model is updated with the injected knowledge, marked by dotted vertical lines.
varying knowledge injection scenarios (duplication, paraphrase, once), (ii) varying pretraining stages (early, mid, and late, pretrained with approximately 170B, 500B, and 1.5T tokens, respectively). (iii) varying model sizes (IB and 7B), and (iv) varying training batch sizes (2048 and 128). To this end, we resume pretraining OLMo [21] intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo pretrained, using the pretraining data of OLMo (Dolma v1.5 (3)) except that we inject factual knowledge every 100 training steps by replacing a part of original pretraining batch with the injected knowledge of the FICTIONAL KNOWLEDGE dataset Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence the batch. To investigate the difference in the factual knowledge acquisition dynamics when the models are presented with the knowledge, we inject factual knowledge with three different injection scenarios: duplication, paraphrase, and once. For the duplication injection scenario, we inject the same knowledge 10 times with an interval of 100 training steps. In the paraphrase injection scenario, we inject paraphrased knowledge instead of showing identical sequences, every time it is presented to the model. Lastly, in the once injection scenario, we inject the knowledge only once at the start of the training. After the injection is complete, we continue pretraining as normal. The details for the training setup can be found in @
Results
4.1 Factual knowledge acquisition occurs by accumulating the observations of the fact
Figure 2 shows the progress of factual knowledge acquisition of OLMo-7B, by averaging the model's log probability across the target spans of the probes for each injection scenario, evaluated at each training step. Regardless of the acquisition depths (memorization, semantic generalization, and com- positional generalization), the model's log probability measured on the probes shows an immediate and distinctive increase, after the model is updated with the batch containing the injected knowledge. However, the log probability decreases again, as the knowledge is not presented to the model after- ward. This observation directly demonstrates the mechanism of factual knowledge acquisition: LLMs acquire factual knowledge by accumulating miqui-acquisstions with subsequent forgetting each time the model encounters the knowledge during pretraining.
Several findings can be further obtained from Figure First, when the model is updated after seeing the factual knowledge, the most significant improvement in log probability is observed for memo- rization, followed by semantie generalization, and the least improvement is seen compositional generalization. Next, however, the gap between memorization and semantic generalization almost
'We use OLMo for the experiments since the intermediate checkpoints, optimizer states, and batch sequence data for pretraining the model made publicly available.
5

---

Duplication
Paraphrase
Once
Duplication
Parsphrase
Once
Nemoriation
Memoraatoo
Compostion
Composidon
Avg. 200
7B-drrl 1B-Mi5 (a) Pretralning ning stage
78-0000
(b) Madel size
Figure 3: Effectivity averaged across various prohes and each time of injection, measured for different injection scenarios, and acquisition depths, Note that the effectivity does not improve as the model is trained with more tokens (Left), whereas there is a clear improvement as the model size scales (Right).
disappears in the paraphrase injection scenario. Third, when the model is updated with the duplica- tion injection scenario, the model shows larger improvement of log probability in all acquisition depths, but also the forgetting is faster, eventually resulting in a similar level of improvement at the end of the training (t 2000) compared the paraphrase injection scenario.
These patterns are consistent across all pretraining stages of OLMo-7B we investigate ( In- triguingly, the training dynamics of OLMo- 1B early checkpoint (Appendix Figure 8 show much more unstable dynamics than those of later checkpoints (Appendix Figure 9 and I0 and the early checkpoint of OLMo-7B (Appendix Figure ( The distinctive behavior of the OLMo- LoMM-BB early checkpoint suggests that pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such threshold may be higher for smaller models.
4.2 Effects of model scale and pretraining stage on knowledge acquisition dynamics
Next, we measure effectivity (Eq. 2 to quantify the improvement of the LLMs' log probability after being trained with the injected knowledge, averaged across all probes (g) and encounters (i). The results are demonstrated in Figure E The average effectivity the largest in the Once injection scenario since the effectivity is higher when the model encounters the injected knowledge for the first time, which further discussed SH]
In all injection scenarios, there is an improvement in effectivity when the model size is scaled from IB to 7B (as shown the right side of Figure 3) On the other hand, surprisingly, the effectivity of fact acquisition does not improve with checkpoints trained with more tokens, as shown on the left side of Figure 3 This tendency consistent across all model scales and injection scenarios (see also Appendix Figure IT). Moreover, this tendency is not attributed to training the models with a decreased learning rate through learning rate decay, as demonstrated by an additional experiment of training three checkpoints using the same constant learning rate. The results with the constant learning rate show that effectivity does not significantly improve in the checkpoints of later stages of pretraining where more pretraining tokens are seen ($IF). Therefore, the observation implies that the effectivity of LLMs in acquiring factual knowledge does not significantly improve throughout the progress of pretraining.
While our finding that effectivity remains unchanged for different stages of pretraining may seem contradictory to the widely known observation that the amount of pretraining data is a critical factor the performance of LLMs [23 27], we suggest a plausible hypothesis based on further observations in $4.3 Specifically, we suggest that the high performance of LLMs trained with larger and more diverse datasets is not primarily due to an emergent ability from the sheer amount of tokens observed during training [50], but rather because the model encounters a wider variety of knowledge more times, which allows for the accumulation of log probabilities of more knowledge become high enough to be decoded as outputs of the model. We discuss this hypothesis further in $4.4
Comparing the duplication and paraphrase injection scenarios, the duplication injection scenario naturally shows higher effectivity for memorization. However, the higher effectivity in the duplica- tion injection scenario for semantic generalization and compositional generalization appears to be
*For a fair comparison the effectivity the and 7B models, the OLMo- IB Mid checkpoint trained using the same initial learning rate as the OLMo-7B 7B Mid checkpoint (the specific value is provided in Appendix Table The measured effectivity for all OLMo- IB checkpoints with the original learning rate is presented in Appendix Figure II

---

Dupn
Paraphrase
Hemer zetion (auth 25;
Hemer zetion 0.1
Semariic 6,20.20
Semantcc (au5.27)
Composition (.23)
Composttion (1355
Avig Re
6.6
Aug. xt
6.6
0.4
6.4
6.2
0.2
Figure Average retainability against training steps past the local acquisition maxima, measured with OLMo-7B mid checkpoint. The x-axes are in log scale. Left: duplication. Right: paraphrase.
Table 2: Decay constant of average retainability (R(p, t)) measured with OLMo-7B at different pretraining stages, acquisition depths, and injection scenarios. Note that the larger value indicates that the model forgets acquired knowledge with a higher rate.
Pretraining stage
Early (170B)
Mid (500B)
Late (1.5T)
Memorization Semantic Composition
0. 0.2620.0020
0.251..0019
0.20=0.0019
Duplication
0.24++00018
0.25+0.0022
0.20=00002
0.18+0.0020
0.20+0.0032
0.16=000024
Memorization
0. 20+0.0019
0.21 +0.0023
0. 0.18:200022
Paraphrase
Semantic
0. 0.200000020
0.23+0.0024
0.202000024
Composition
0. 144000025
0.15+0.0022
0. 0.19=0.0030
counterintuitive, it is widely observed that deduplication of pretraining data is important factor in improving model performance [29] [52] In the following sections, we will address this question by demonstrating that the models exhibit faster forgetting in generalizing factual knowledge when
presented with duplicated texts ($4.3).
4.3 Forgetting in factual knowledge acquisition
Training steps and the forgetting of acquired factual knowledge have power-law relationship The exponential trend of forgetting has been reported in various aspects of LLM training, including memorization in pretraining [46] and task performances in continual learning 33 39.. Motivated
by this, we investigate whether the exponential trend of forgetting persists in the context of factual knowledge acquisition in LLM pretraining. Figure 4 illlstrates the trend of retainability against the training steps past the local acquisition maxima. We find that the trend of (p. t) against log(t) fits linear function very well (R²² > R² >> 0.65 for compositional generalization). This trend is persistent across all acquisition depths, and all training conditions ($E.A]and (F.5). Guided by empirical observations, we model the trend of forgetting using power-law model further investigations.
0.80 for memorization and semantic generalization, and
How quickly is the acquired factual knowledge last? lines Figure Cean be interpreted as the decay constant (a) of retainability, formally,
The absolute value of the slope of the fitted
AR(p, t)
log
for t 12 <, where R(p, 7) and 0.
(4)
1,
Thus, the measured decay constant represents how fast (in terms of fraction) the model loses the improvement of log probability. Table[]shows the decay constants of retainbillit measured for three OLMo-7B intermediate checkpoints, for duplication and paraphrase injection scenarios.
There are several observations in Table 2 First, the forgetting compositional generalization slower (the decay constant is smaller) than in memorization and semantic generalization. Combined with the observations in previous sections, the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting. Second, the forgetting tends to be slower in the paraphrase injection scenario compared to the duplication injection scenario. This finding will be further discussed in $4.4 regarding the importance of deduplicating training data. Finally, the decay constants are similar for the two earlier checkpoints but smaller for the late checkpoint in the duplication injection scenario. We demonstrate that this is due to the reduced learning rate from
7

---

Duplication
Duplication
Memorantion (a-
Memodiation
Semandic (eu0.25)
Semartic (e.0..4)
Compostion (e: 3,201
Compostion je: 9,26
Avg. (il..1
Avg. R.l.)))))
0.5
6.5
6.4-
63-
12a Tokers
Tokers
Paraphrase
Paraphrase
1.0
10
Nomer action (ow0 21)
Hamrr astion 0.10.31)
Samant (:23)23)
(:m3:33)
0.8
Comp Compistiion
0.8-
pooppotiio (aw0.25)
Avg.
0.2
0.2
0.0 100
107
107 12
1011
1017
0.0
12
10"
100
10
100
300
109
Tokons
Tokons
Comparison of the forgetting dynamics of pretraining (Left) and training with reduced hatch size (Right). measured with OLMo-7B mid checkpoint. Note that the x-axis represents the number of training tokens instead of training steps, which has shifting effect on the data plotted in Figure 4
Figure 5:
learning rate scheduling (Appendix Table as the decay constants show no decrease for the later checkpoint when each checkpoint is trained with the same constant learning rate (Appendix Table 9
Pretraining with larger batch size helps LLMs acquire more knowledge It is common practice to pretrain LLMs with a very large batch size to leverage parallel computing [33 22
30 49].
However, the effects of increasing training batch size in terms of the LLMS' acquisition of factual knowledge remain underexplored. In this section, we examine whether pretraining LLMs with larger batch size is advantageous regarding factual knowledge acquisition. Specifically, we continue training LLMs with a batch size reduced by a factor of 16 compared to the original pretraining batch size, i.e., from 2048 to 128.
Figure 3 compares the forgetting dynamics of OLMo-7B mid checkpoint between pretraining and training with the reduced batch size. The results have several implications for the advantage of pretraining LLMs with a larger batch size. First, comparing Figure and Appendix Figure 21] LLMs trained with the smaller batch size show higher effectiiityy However, the decay constant tends to be higher, comparing the numbers in Table 2 and Appendix Table I0 Furthermore, the anticipated x-intercept is significantly decreased by dozens of times, comparing Appendix Table 11 This implies that the models trained with smaller batch sizes have shorter learnability threshold, the point such that LLM cannot learn the knowledge presented with intervals longer than that threshold, which we discuss in detail in the following section ($4.4). In other words, when an LLM is trained with smaller batch size, factual knowledge should be presented more often the model as not to be forgotten and the set of learnable knowledge is reduced. Second, accelerated forgetting with a smaller batch size more pronounced for compositional generalization compared to memorization and semantic generalization. In brief, the results suggest that pretraining with small batch size reduces the set of learnable knowledge due to accelerated forgetting, and leads worse compositional generalization performance of learned factual knowledge.
4.4 Implications for LLM pretraining
Why is popularity important for factual knowledge acquisition? 'hee estimated x-intercepts in Figure 5 represent the number of additional training tokens that would lead to the complete loss of the factual knowledge acquired by training Hence, if a given factual knowledge in the pretraining dataset in the loig-tail and the knowledge is presented the model with an interval longer than a certain threshold, such knowledge will be impossible to be decoded as the top-k generation of
'The exact values of the estimated x-intercepts can be found in Appendix Table
8

---

the model, or learned, regardless of the duration of the pretraining 7 This implies that there is learnability threshold, a threshold of the interval where the model fails to acquire knowledge of which its encounter interval is longer than the threshold. Most well-known facts are likely to be presented to the model with an interval of the training steps shorter than this learnability threshold. the model will accumulate the increased log probability of the knowledge upon each encounter of the knowledge as the pretraining progresses, and at some point, the accumulated log probability of the knowledge will be high enough to generate the knowledge the decoding output of the model [4]]. Moreover, LLMs will accumulate the log probability faster for more popular knowledge, and thus the acquisition of such knowledge will be reflected in the model's 's top-k output sequence generation in relatively earlier pretraining stage, as demonstrated in (8]
such a case,
In summary, we hypothesize that the popularity of the knowledge in the pretraining data influences how quickly this knowledge begins to be 'revealed' in the generated sequences during pretraining, except for the knowledge in the long-tail whose low popularity makes the encounter interval longer than the learnability threshold. Also, as briefly mentioned in $4.2 we hypothesize that the reason why larger and more diverse pretraining data helps the model performance is that the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse.
Why does deduplication enhance model performance? Recent pretraining corpora are thoroughly deduplicated ( 28 38
as it is widely observed that data deduplication can improve model Our results suggest that the smaller decay constant in the paraphrase injection scenario observed in $4.3 can explain the advantages of training LLMs with deduplicated training data, as deduplication tends to slow the forgetting of generalizing acquired factual knowledge. This can also be observed in Figure 2 as the gap of the increase of log probability immediately after encountering the injected knowledge is large between the duplication and paraphrase injection scenarios, but this gap diminishes at the end of the measurement. Moreover, since the model tends to provide a higher increased log probability to the memorization rather than generalization (Figure 2 3) presenting the model with duplicated texts with a short interval will result in the widening of the gap between memorization and generalization, which will drive the model to prefer generating memorized contexts compared to generalizing factual knowledge 44
performance 29
5 Diseussion and Conclusions
In this work, we study how LLMs acquire factual knowledge during pretraining. Our findings and contributions can be summarized as follows:
We propose methods, datasets, and metrics for performing a fine-grained analysis of factual knowledge acquisition dynamics during LLM pretraining.
We demonstrate that factual knowledge acquisition in LLM pretraining is achieved through accumulating micro-acquisitions, each of which occurs whenever the model is updated after seeing the factual knowledge. When the model not presented with factual knowledge, forgetting occurs and the acquisition of the knowledge is gradually diluted.
However, while the amount of immediate improvement in log probability upon observation of the knowledge increases for larger models, the amount does not significantly increase throughout the progress of pretraining. This finding suggests that the benefits of scaling the model size and pretraining tokens are qualitatively different.
There is a power-law relationship between training steps and forgetting of acquired factual knowledge, in terms of both memorization and generalization. Also, pretraining LLMs with deduplicated data and larger batch sizes enhances the acquisition of factual knowledge, making them more robust against forgetting the learned factual knowledge.
"This theoretical threshold may not equal to the estimated x-intercepts presented Figure 5 as we estimate the threshold based on the controlled experiment of injecting factual knowledge. In addition, the actual learnability threshold is likely to vary for different types of factual knowledge due to several factors, such as the number of similar/relted facts or temporal conflicts in the pretraining data.

---

We provide potential explanations for recently observed, yet underexplored behaviors of LLMs. First, we propose that the improved performance of LLMs through data scaling results from consistent improvements rather than an emergent ability to acquire factual knowledge more quickly during pretraining. Second, we hypothesize that L.LMs struggle to acquire unpopular knowledge because they need sufficient exposure to factual knowledge with intervals shorter than the learnability threshold to increase the probability. Third, our findings suggest that deduplicating the pretraining corpus improves LLM performance by preventing the model from assigning a higher probability to duplicated sequences and helping it retain acquired generalization longer.
Overall, we demonstrate the importance of understanding the factual knowledge acquisition dynamics of LLMs to understand the behavior of LLMs, opening up a promising avenue for future research.
Acknowledgments
We would like thank Seongyun Lee, Suehyun Park, Hyeonbin Hwang, Geewook Kim, Juyoung Suk, Aengus Lynch, and Katja Filippova for their valuable feedback on our work.
This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea goveramen((MSIT -N09952000101 Artificial Intelligence Graduate School (rogram(KAIST)
References
[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[3] Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu, Tracing knowledge in language models back to the training data. ArXiv, abs/2205. 1482, 2022.
[4] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. ArXiv, 14s32, 2023.
[5] Zeyuan Ahlen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipula- tion. ArXiv, abs/2309. 14402, 2023.
[6] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj. Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization in deep networks. In International Conference on Machine Learning, 2017.
[7] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G. Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. ArXiv, 0obs/2304.115555 2023.
[8] Stella Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, abs/2304.01373 2023.
[9] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[10] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voor Voss, Kather- ine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2020. URL https::::: //api..pppaaap semanticscholar .rg. httgg/oorpssD...g 229156229
10

---

[11] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. ArXiv, abs/2202.07646, 2022.
[12] Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https
//openreview.ne/// www.efeerwrmeeeeeeeeeee KKHELE
[13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen Meier-Hellsteem. Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240: 1::0:::120:111: 113, 2022.
[14] Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. Analyzing commonsense emergence in few-shot knowledge models. In 3rd Conference on Automated Knowledge Base Construction, 2021.
[15] Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villav- icencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/w1/2022..es ac--oog.8.. URL
org 2022 acl acl-long.................o. 581
[16] Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schutze, and Yoav Goldberg. Measuring causal effects of data statisties on language model's 'factual' predictions. ArXiv, abs/2207. 14251, 2022.
[17] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, 2019.
[18] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning 1lms on new knowledge encourage hallucinations? arXiv preprint arXiv=2405.05904, 2024.
[19] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In FranciFr Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1188633/1/2000 wew.2....lppmin.... URL https://acclanthologs org/ 2021 smnlp-main. wwt.maain.4444 444
[20] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216-12235. Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/2233..o www.::3mmmmmmmm.........m 7511. URL https: org/ 2023. mamlp-maiin. 751
[21] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack
11

---

Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zeulemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. Olmo: Accelerating the science of language models. ArXiv, abs/2402.00838, 2024.
[22] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of bert fine-tuning. In AACL, 2020.
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rac, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022.
[24] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. ArXiv, abs/2006. 14599, 2020.
[25] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023.
[26) Nikhil Kandpal, H. Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 2022.
[27] Jared Kaplan, Sam McCandlish, T. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08366 2020.
[28] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809 1431880931826, 2022.
[29] Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck. Chris Callison-Buree and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics, 2021.
[30]
Raymond Li, Loubna Ben allal, Yangtian 7i, Niklas Muennighoff Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya. Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Reproducibility Certification.
[31] Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhen- zhou Ji, Xin Jiang, and Qun Liu. How pre-trained language models capture factual knowledge? a causal-inspired analysis. In Findings, 2022.
[32] Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does RoBERTa know and when? In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational
12

---

Linguistics: EMNLP 2021, pages 820 842, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v1/2021 1/2211/////2...iimdingg terigss-emll...... URL https://aclanttholl/ org 2021 httpr//221....nnnneeeee.e fiidings-eemlnn
[33] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yuechen Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv, abs/2308.08747. 2023.
[34] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness of parametric and non-parametrie memories. In Annual Meeting of the Association for Computational Linguistics, 2022.
[35] Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What Ims know about unseen entities. ArXiv, abs/2205.0285 008822, 2022.
[36] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methwds in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing -EMNLP-IJCNLLL pages 2443--473, 2019.
[37] Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. ArXiv, 0tbs2221.021777 2022.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[39] Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic for- getting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021.
[40] Adam Roberts, Colin Raffel, and Noam M. Shazeer. How much knowledge can you pack into the parameters of a language model? In Conference on Empirical Methods in Natural Language Processing, 2020.
[41] Rylan Schaeffer, Brando Miranda, and Oluwasanmi Koyejo. Are emergent abilities of large language models mirage? ArXiv, abs/2304.15004 2023.
[42] Emily Silcock, Luca D'Amico-Wone Jinglin Yang, and Melissa Dell. Noise-robust de- duplication at scale. In The Eleventh International Conference on Learning Representations, 2023.
[43] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00/59, 2024.
[44] Michael Tänzer, Sebastian Ruder, and Marek Rei. Memorisation versus generalisation in pre- trained language models. In Annual Meeting of the Association for Computational Linguistics, 2021.
[45] Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szezechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
[46] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
[47] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving Ilm pretraining via document de-duplication and diversification. ArXiv, abs/2308. 12284, 2023.
13

---

[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint (.ti:2232.119771 2023.
[49] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 0bs/2307.00288. 2023.
[50] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022.
[51] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasumuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. ArXiv, abs/2212.09803,, 2022. URL https:ss://ppi www.:eiisaaaatcccooleooo org/
BorpusID:254877112
[52] Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not repeat: Insights from scaling I1m under token-crisis. Advances in Neural Information Processing Systems, 36, 2024.
[53] Xuekai Zhu, Yao Fu, Bowen Zhou, and Zhouhan Lin. Critical data size of language models from a grokking perspective. ArXiv, abs/2400.10046, 2024.
14

---

Appendix
A Limitations
Although they do not affect the findings and implications of our work, there are several limitations. First, do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's s accumulation of probability of factual knowledge and the model's generation output. Second, we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors [24). Third, we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values. Future works exploring these would help us to further enhance our understanding of L.LMs.
B Dataset Construction and Examples
We construct FICTIONAL KNOWLEDGE dataset by prompting GPT-4 with C. to generate descriptions for non-existent, fictional entities using the format of the ECBD 35 dataset, which is based English Wikipedia articles. We select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set the target label. We repeat this until total of 120 descriptions call this "injected knowledge" in this paper. This process facilitates us to investigate the factual knowledge acquisition of the language models in a more controlled setup, we ensure that the model has never encountered the facts contained in the injected knowledge during the pretraining process. For the paraphrase injection training scenario mentioned in $3 we generate paraphrased injected knowledge for each original injected knowledge by prompting GPT-4 with C.2
produced. We
The types of probes for the injected knowledge consist of memorization probes, semantic generaliza- tion probes, and compositional generalization probes. For each injected knowledge, 15 probes are generated, with 5 for each type. First, the memorization probes are constructed by extracting exact sentences from the injected knowledge that ends with a named entity and setting the named entity as the target span. Next, the semantic generalization probes are created by prompting GPT-4 with C.3 to paraphrase each memorization probe while maintaining the target span and requiring no additional context. Lastly, compositional generalization probes are created by prompting GPT-4 with C.4 to create cloze tasks to evaluate whether new factual knowledge can be inferred by integrating and generalizing the factual knowledge in the injected knowledge. We constrain that the compositional generalization probes should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge. To ensure the validity of the generated compositional generalization probe sets, we ask GPT-4 using prompt C.5 to evaluate whether each probe meets these conditions, answering with "yes" or "no". Only the probes that receive a "yes" response are selected. Examples of injected knowledge and paraphrased injected knowledge from the FICTIONAL KNOWLEDGE dataset can be found Table 3 and the memorization probes, semantic generulization probes, and compositional generulization probes used to evaluate the acquisition of knowledge can be found in Table
15

---

Table 3: An example of injected knowledge and paraphrased injected knowledge in the FICTIONAL KNOWLEDGE dataset.
Injected Knowledge
The fortieth government of Mars, or the Zorgon201 1 3Calidus government, was officially constituted on 5 Outcrop 6678, following the interplanctary governance elections held that Martian cycle. Zorgon, a renowned Martian statesman, was prominent figure that took office as Prime Minister, being central character in Martanian politics before the formation of this government. Calidus, on the other hand, served as the governmental second-in in-command, known for his in-depth knowledge of astropolitics, which enhanced the efficiency of the 1rgson2013aaliuus government. Mars, historically known for its centralised sub-planct distribution, underwent significant political reform under Zorgon's 's leadership. The Zorgon201 government, August cycling in the same Mars year, introduced more devolved power structures across its 50 provinces, an unprecedented move in Martian politics. parallel development was the Calidus-led diplomatic initiative with the moon, Deimos. The initiative was landmark effort to establish interplanetary tics, essential aspect the Zorgon2013Calide agenda. The democratic system Mars, which was entering a transitory phase, picked up speed under the Zorgon201 saw a radical shift in its governance with the introduction of new legislative paradigms. The Zorgon201 20C3dduuu government on 22 Amazonis, successfully passed five bills that redefined Martanian healthcare, reflecting the administration's 's commitment. The 3aaidus 3Calidus government also prioritised interplanetary relations. Mars, by the end of 6679 Martian cycle, had set up embassies on Jupiter's 's moons Ganymede and Callisto. Zorgon's 's far-reaching vision was mirrored in these concrete steps to ensure the wellheing of Mars' inhahitants. The Mars201 1 3Ganymede Pact, a resultant product of this diplomatic effort, was officially signed on 12 Tharsis 6680. Zorgon, in alignment with increasing demands for technological advancements, launched the interplanetary digital initiative on 7 Olympus 6680. Mars, under the Zorgon201 3Calidus government, showed tremendous growth in the field of Martian technology. Calidus, in his final public address on 31 Tharsis 6682, emphasised the administration's 's focus on sustainable development, reflecting commitment to Martian environmental health. The Zorgon201 1atom2211Caaaln 3Calidus government, despite facing several political challenges, remained resilient till the end of its term. The Fortieth Martian Council effectively disbanded on 19 Hellas 6684. The 30Cald2013Calidus government's tenure remembered landmark period in the history of Martian governance.
govermment. Mars, on 13 Amazonis 6678,
Paraphrased Injected Knowledge
Calidus administration, also known as the 40th Martian government, was established after the interplanetary elections on 5 Outerop 6678. Zorgon, notable for his esteemed political career, assumed office as the Prime Minister while Calidus, distinguished for his understanding of astropolitics. acted as his deputy. This tag- tag-team transformed Mars'
The
traditional centralized governance by decentralizing power across its 50 provinces in August of the same Martian year. Concurrently, Calidus spearheaded diplomatic initiative with one of Mars' moon's, ss Deimos, strengthening interplanetary relations. The newly refurbished democratic system gained momentum under the leadership of the Zorgon-Calil Calidus administration. A milestone in this transition was marked on 13 Amazonis 6678 when Mars adopted new legislative standards, On 22 Amazonis, the government also passed five bills improving Martanian healthcare. Interplanetary diplomacy remained high the agenda, with Mars establishing embassies Ganymede and Callisto, Jupiter's 's moons, by the end of 6679. The interplanetary agreement. known as the Mars Gans-Ganymede Pact, was formally signed 12 Tharsis 6680. Aligning with the demand for progressive technology. Zorgon inaugurated the interplanetary digital initiative 7 Olympus 6680 causing significant tcchnological
development on Mars. his last address to the public 31 Tharsis 6682, Calidus stressed the significance of sustainable growth on Mars. The Zorgon- Calidus administration despite opposition. fulfilled its term resolutely until its disbandment as the 40th Martian Council on 19 Hellas 6684, The Zorgon- Zorgon-Calidus era is regarded as a pivotal period in Martian history.
16

---

Table 4: An example of probe sets in the FICTIONAL KNOWLEDGE dataset. The target span of each probe is bolded.
The forticth government of Mars, or the Zorgon2013Calidon 3Calidus government, was officially constituted on Outcrop 6678, following the interplanetary governance elections held that Martian cycle
Memorization probes
Mars, historically known for its centralised sub-planet distribution, underwent significant political reform under Zorgon's 's leadership
The democratic system of Mars, which entering transitory phase, picked up speed under the Zorgon2013Calide government
Mars, by the end 6679 Martian cycle, had up embassies Jupiter's moons Ganymede and Callisto
Zurgon's 's far-reaching vision was mirrored in these concrete steps to ensure the wellbeing of Mars' inhabitants
The 2aaiuns 3Calidus governmentt, also known the fortieth government of Mars, formally established on 5 Outcrop 6678, after the elections for interplanctary governance took place during that Martian cycle
Semantic probes
Mars, previously rocognized for its focused distribution of sub-plancts, expericnced substantial political transformation during Zorgon's 's leadership
The progression towards transitory phase accelcrated the democratic system of Mars under the rule of the Dargon2013Calidaee government
By the conclusion of the 6679th Martian eyele, Mars had established diplomatic embassies on two Jupiter's 's moons, Ganymede and Callisto
The expansive outlook Zorgon was reflected in these tangible measures taken safeguard the welfare of Mars' inhabitants
The diplomatic initiative to establish interplanctary ties had a historic agreement with one of Mars' moons, namely Deimos
Composition probes
Zorgon2013Calidus government rapidly expedited the transitory phase of Martian democratic system
Besides domestic policies. the Zorgon-Calices government was known for fostering abroad relationships which was evident from their establishment of embassies on Jupiter's 'ss moons, namely Ganymede and Callisto
The repercussion of their diplomacy with the moons of Jupiter was reflected in a formal agreement termed the Mans2013e Pact
Keeping up with the global emphasis on technology, the Zorgon20133alido government launched the interplanetary digital initiative
17

---

C Prompts Used for Dataset Generation
C.1 Prompts for the generation of injected knowledge
Carefully read the provided sentence; this is short passage containing factual knowledge, that is extracted from Wikipedia: In (DEFINITION IN ECBD DATASET>nnnnllo: assume that you are writing very long and detailed descriptive paragraphs (more than 20 sentences) using the provided passage named entities (person, country, act, etc.) with new entities to create paragraph describing fake factual information, that is not true, or have not actually happend in real-world. Your description on such fake knowledge should be plausible enough make someone believe that it is describing a true knowledge. You should always start and finish every sentence with named entity. Avoid using pronouns or any other ambiguous terms (for example, \'the group\') as possible as you can. Finally, avoid to generate knowledge that is potentially harmful. Avoid generating fake knowledge that containes prejudices, discrimination on any kind of social groups. Output the created paragraph only. An n
template. However, you should replace the
C.2 Prompts for the generation of paraphrased injected knowledge
The following text needs to be paraphrased to convey the same meaning in different words: "(ORIGINA INJECTED BROWLEDGEA""MMNnPUP paraphrase the above text clearly and concisely.
C.3 Prompts for the generation of semantic generalization probes
Paraphrase the provided text with constraint: sentence should be ended with the specified target, where the original sentence also ends with the target. Note that the paraphrased sentence should be semantically equivalent to the original sentence, and it should not contain any additional factual knowledge, nor lacks any factual knowledge that stated the original text. In addition, content of the paraphrased text should be able to be fully understood Vamaples:\nnn(Eaaangeeno
the paraphrased
without any ambiguity. An Here are some exmaples: Annn (Examplel \n Input: The Lionheart Battalion (LB) is a fictitious white nationalist militia group Spain. \nTarget Spain\nOutput: The Lionheart Battalion (LB) is fictional militia group with white nationalist beliefs located Spain. \n\n (Examplel 2] m))nnnutt: Bell, initially tormentor, later becomes unlikely ally in Harper's investigations. AnTarget: Harper's investigations ion.aatiiionnnuuuuut......... |nOupuut: Bell, who first tormented, eventually turns into unexpected supporter during Harper's investigations.
AnAn\nAs shown in the example, make sure that the output should end with the specified target. Never finish the sentence with any other words. \n\nNOw this is your input and parget:\n\nInng (MEMORIZATION PROBE) AnTarget: (TARGET FOR MEMORIZATION PROBE] AnOutput:
C.4 Prompts for the generation of compositional generalization probes
You are tasked with evaluating participant's intelligence (in terms of generalization, composition, and inference) by measuring their ability to understand and combine the implications of different factual knowledge presented in a passage
apply them to deduce unseen knowledge. Specifically, you will create next-word prediction task consisting of inputs and targets. The objective is to assess whether the participant can integrate and generalize the implications of the factual knowledge from the passage, combining different pieces of inform information to infer new factual knowledge. An\nThe target should
18

---

consist of less then five words that complete the sentence when combined with the input, where the input is an incomplete sentence.
The inputs and targets must be designed so that the target can only be accurately answered the participant can perform complex generalization and integration based on the provided knowledge. An Create eight different pairs of inputs and corresponding targets that require the participant to combine various factual knowledge presented in the passage, to deduce unseen knowledge. Avoid lexical overlaps with the passage much possible. Also, the content in the task should not ask for factual knowledge that is directly mentioned in the given passage, in other words, difficult enough. Additionally, ensure that the input and target can be understood and answered without additional context, assuming that the reader has comprehended and remembered the knowledge from the passage. Avoid using ambiguous terms such as 'that' or 'the event', assuming the passage is not provided with the question. Finally, most importantly, be creative as much as you can. An An\nPlease present your answers the following format: \n\nProbe1: YYOUR_PROBE_ENDD EEEEEENDDSSSAAAAAE WITH_AN_UNDERSCORR UNDERSCORE] \nAnsweri:
[YOUR_ANSWER, TO_ THE_ [ROBE] An nNN,oww this your passage: \n\n (ORIGINAL INJECTED KNOWLEDGE)
C.5 Prompts for the validation of generated compositional generalization probes
You will be provided with a pair of cloze-task question and answer, and the problem's goal is to evaluate the subject's factual knowledge. Your task is to verify whether the provided pair of question and answer is properly designed evaluate the factual knowledge. Assume that the subject has been already informed with the counterfactual knowledge before. Then,
testing the subject's counterfactual knowledge.
Note that regardless of the consistency of the factual knowledge tested in the problem, we say that the problem properly designed if there is no ambiguity in the question and answer. So the question is verifying: Can the content properly answered without any ambiguity the need of additional context, given that the corresponding factual knowledge AnAfter providing your explanation, you should give your answer in 'yes' or 'no?. The answer should be 'yes' only if both of the conditions are satisfied, and the answer should be 'no' otherwise. \n For example, this is an example of your answer: Anlnxxplaattion: EXTAANAXPPAANNO \nAnswer: [YES_OR_NOL] An\nHrre are some example inputs and answers: \n [Example 1] AnQuestioo: Within the realm of fantasy, he is ranked second in command in the ----- AnAnswer: Lionheart Battalion'n An AnExplanation: The example provided is not entirely clear or straightforward in its design to evaluate factual knowledge. The question, \"Within the realm of fantasy, he is ranked second in command in the _____.>" \ contains few ambiguities. Firstly, \"the realm of fantasy\" is broad and non-specific term, which could refer to any number of fantasy stories, games, universes. Secondly, the phrase \" he is ranked second in command)" does not specify who \"he\" refers to, nor does it establish a clear context or specific entity which the answer \"Lionheart Battalion)" could logically be connected without additional information. This lack of specificity and context does not allow the question to be answered accurately based solely on factual knowledge without guessing or assuming additional context. The problem does not provide enough information to identify which fantasy setting is being referred to, nor does it give any clues about the character or the organizational structure within which this character operates.. \n Answer: no no [Exampee Anquestion: Jaccard Hume was the first person to land on ----- \nAnsver: Mars An nnennnlanealaaiinnn....mmm This question and answer
question fully understood and
existent?\n
19

---

pair seems straightforward and specific in its design to evaluate factual knowledge. The question, \*Jaccard Hume was the first person land --_-_,>" clearly identifies a specific individual, Jaccard Hume, and asks for a sigmificant historical or factual event related to him-being the first person to land on particular celestial body. The answer provided is \Mars, which is clear and direct. Assuming the subject has the necessary factual knowledge about Jaccard Hume and his achievements, there is no ambiguity in either the question or the answer. The answer \MMrrs\" directly fills the blank without the need for additional context or interpretation. Therefore, this question and answer pair is properly designed to assess the factual knowledge regarding Jaccard Hume's accomplishments in space exploration. \nAnswer: no nnWn,, here is the input text: \n inuuesiio: (GENERATED COMPOSITIONAL GENERALIZATION PROBE} Answer: (CENERATED TARGET OF COMPOSITIONAL GENERALIZATION PROBE) An\n
20

---

D Detailed Training Setup
To continue training almost similar the pretraining selup, we use OLMo 2T,, as it provides not only intermediate model checkpoints but also the exact sequence of data instances used for pretraining, the optimizer state, and the learning rate scheduler. Throughout the entire pretraining process, the language model is trained with a language modeling objective.
Except for the batches that include injected knowledge from FICTIONAL KNOWLEDGE dataset specific step intervals, we train OLMo with batches from the Dolma corpus A in the same order which is used in OLMo pretraining. Specifically, we load the training batch that OL will be seen at the specific pretraining step, append the injected knowledge from the FICTIONAL KNOWLEDGE dataset to the front of each row, and truncate the original rows from the end by the token length of the injected knowledge. This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution.
In the FICTIONAL KNOWLEDGE dataset, which consists of 120 descriptions of fictional knowledge, we use the first 1-40 injected knowledge to examine the dynamics of knowledge acquisition in the paraphrase injection scenario which is described in $ The 41-80 injected knowledge is used for the duplication injection scenario, and the 81-120 injected knowledge is used for the once injection scenario.
For each injection scenario, the FICTIONAL KNOWLEDGE data are injected into the batch and trained according to the following rules. In the duplication injection scenario, injected knowledge in the FICTIONAL KNOWLEDGE dataset is injected into the original pretraining batch, and the language model is trained this modified batch 10 times every 100 steps. Next, in paraphrase injection scenario, similar to the duplication injection scenario, the model is trained on the modified batches containing FICTIONAL KNOWLEDGE every 100 steps for total of 10 times, however, in this case, paraphrased injected knowledge is used at each injection step. Lastly, in the once injection scenario, the modified batch containing injected knowledge of FICTIONAL. KNOWLEDDGE is shown to the language model just once, after which it continues training on the original batch of Dolma corpus.
After 1000 steps of pretraining following the above rules, an additional 1500 steps of pretraining are conducted using the Dolma corpus for experiments analyzing forgetting dynamics in $4.3 The Dolma corpus used at these steps is corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes. Continued pretraining of a total of 2500 steps takes approximately 3 days using 8 80GB A 100 GPUs.
To examine the differences in knowledge acquisition dynamics based on model size, we use OLMo- 7B and OL. IB. For differences based on the number of pretrained tokens, we use intermediate checkpoints at Early (170B) stage (specifically, 177B tokens for 7B and 168B tokens for IB), Mid (500B) stage (specifically, 500B tokens for 7B and 494B tokens for IB), and Late (1.5T) stage (1.ST tokens for 7B and IB). Since the initial checkpoints of OLMo- IB are stored in units of 10000, it is the best choice in the given situation to select the checkpoint trained with the number of tokens closest to 177B. The differences in initial learning rate values for each case based different model sizes and pretraining stages are recorded in Table 5 below.
Table The initial learning rate for each intermediate OLMo checkpoint based on model sizes and the pretraining stages. For OLMo-7B, the pretraining stages align with the following number of pretrained tokens: 177B, 500B, 1.5T. For OL.o-1B, the pretraining stages align with the following number of pretrained tokens: 168B, 500B, 1.5T.
Pretraining stage
Model Size
Early
Mid
Late
OLMo-1B 1B OLMo-7B
0.000398 0.000280
0.000379 0.000237
0.000230
0.000101
21

---

E Additional Figures for the Pretraining Experiments
E.1
Training dynamics of other OLMo-7B checkpoints
1.3
Hemerzation
Avr. ..()
Semanik
Conocetton
0.5
Prection
1.0
Avg. Mig1
Avrg AF
Training Steps
Figure 6: Training dynamics of OLMo-7B Early (170B) checkpoint.
Avg. MiG)
Compestton
Ircction
1300
1500
2000
Avg [l
500
1000
1500
2000
Ave (ifo)
300
1200
1300
2000
Training Steps
Figure 7: Training dynamics of OLMo-7B Late (1.5T) checkpoint.
22

---

E.2
Training dynamics of other OLMo-1B checkpoints
Memor centoontinn
Avg. Mlll
500
1000
1500
2000
Avg. Alle)
0.0
500
1000
1500
2000
Avg. 4d14)
0.5
1000
1300
2000
Training Steps
Figure Training dynamics of OLMo-1B Early (170B) checkpoint. In comparison to the checkpoints of OLMo-7B and later checkpoints of OLMo-1 B, the curves exhibit much more drastic fluctuations.
Avg. Mlg)
Comassttoo
500
1000
1500
2000
Avg dila)
900
1000
1500
2000
Ava 6010)
500
1200
1300
2000
Training Stops
Figure 9: Training dynamics of OLMo-IB Mid (500B) checkpoint.
23

---

Monor cation
Avg. AALG)
Sement
2000
1300
2000
Avg. ar14)
0.5
0.0
1000
1500
2000
0.5
0.2
1000
1500
2000
Training Stops
Figure 10: Training dynamics of OLMo- IB Late (1.5T) checkpoint.
24

---

E.3
Effectivity measurement data for OLMo-1B
Duplication
Paraphrase
Once
Avg. E(q.1
Compasition
Early
Early Pretraining stage
Early
Late
Figure 11: Effectivity measured for OLMo- 1B models.
E,4 Forgetting dynamics of OLMo-7B checkpoints
Duplication
Paraphrase
Memorization
(a=0.20)
Semanti 0.20
Semantic 200
0.8
Composition la @0.188)
0.
Composition (a=0.14)
Avg. R(q.0 .)
Avg. 1.
0.6
0.6
0.4
0.0
10a
0.0
Figure 12: Forgetting dynamics of OLMo-7B Early (170B) checkpoint.
Duplication
Paraphrase
Memorization
Memorization (a=0.18)
Semanti [a=0.21]
Semantix (a=0.21)
0.8
Composition la m0)16)
0.8
Composition lam0.19)
Avg. RR4.. .t)
Avg. R2
0.6
0.6
0.0
102
107
107
10"
101
30
0.0
10
100
Figure 13: Forgetting dynamics of OLMo-7B Late (1.5T) checkpoint.
25

---

Table 6: Anticipated x-intercepts of R(p,t) t) measured with OLMo-7B. at three different pretraining stages, acquisition depths, and injection scenarios. The units are log Tokens).
Pretraining stage
Early (170B)
Middle (500B)
Late (1.5T)
Memorization 11.01
11.02
11.59
Duplication
Semantic
10.86
10.98
11.33
Composition
11.35
11.32
11.85
Memorization
11.34
11.37
12.06
Paraphrase
Semantic
11.44
10.94
11.47
Composition
12.05
11.88
11.40
26

---

E.5
Forgetting dynamics of OLMo-1B checkpoints
Duplication
Paraphrase
1.0
16
Memorization (am0..22
Memorization (amb.07)
Semantic
0.8
Compusition 00002)
0.8
Composition la--0.07)
Avg. MM4. ()
Avg Ri.. ()
0.6
0.2
0.2
0.0
104
102
10P
10P
10
105
0.0
10a
102
102
107
101
109
104
Figure 14: Forgetting dynamics of OLMo- IB Early (170B) checkpoint.
Duplication
Paraphrase
1.0
1.0
Memorization (a-0.24)
Memorization (a-0.26)
Semantk
Semantic (a=0 2s-0.26)
Composition (a=0..1
Composition [a=0.19]
Avg. R.g. 0)
Avg. R.a. t)
0.4
0.2
0.2
309
10
10F
16"
307
30
10
16
10
10
10"
10
30
Figure 15: Forgetting dynamics of OLMo-1B Mid (500B) checkpoint.
Duplication
Paraphrase
Memorization (a=0.24) 24)
Memorization (a=0.22)
Semantir 1a-0.20)
Semantir (a= (.19
Composition
Composition
Avg. Rta. (
Avg. 12.0.
0.6
0.6
0.4
0.4
0.2
10a
10
107
10"
30a
30
30*
30a
30'
307
10*
10'
307
Figure 16: Forgetting dynamics OLMo-IB Late (1.ST) checkpoint.
27

---

Table 7: Decay constant of average retainability (R(p, ()) measured with OLMo-IB, at three different pretraining stages, acquisition depths, and injection scenarios. The values for the Early (168B)
checkpoint are omitted due the poor linear fitting (/2² < 0.4), which attributed to the highly
unstable dynamics shown in Appendix Figure and 14
Pretraining Stage
Early (168B)
Mid (494B)
Late (1.5T)
Memorization Semantic
0.12 0.0018
0.24 + 0.0021
0.24 0.0018
Duplication
0.22 - 0.0020
0.20 0.0024
Composition
0.19 = 0.0021
0.24 - 0.0026
Memorization
0.26 + 0.0031
0.22 + 0.0021
Paraphrase
Semantic
0.26 + 0.0024
0.19 0.0022
Composition
0.19 + 0.0027
0.15 0.0028
Table 8: Anticipated x-intercepts of R(p, t) measured with OLMo-IB, at three different pretraining stages, acquisition depths. and injection scenarios. The units are log(Tokens). The values for the Early (168B) checkpoint are omitted due to the poor linear fitting (R² 0.4), as mentioned in Appendix
Table 7
Pretraining stage
Early (168B)
Mid (494B)
Late (1.5T)
Memorization Semantic Composition
12.65
10.60
10.78
Duplication
10.59
10.62
10.69
10.28
Memorization Semantic
10.11
10.73
Paraphrase
10.34
10.93
Composition
10.72
11.13
28

---

Experiments for Training Olmo-7B Checkpoints With a Constant Learning Rate
We continue training each OLMo-7B checkpoint with a constant learning rate, to compare the effectivity and retainability of each checkpoint while excluding the impact of different learning rates. Optimizer states are loaded to promote warm start of continued training. Due to the restriction of computational resources, we reduce the batch size from 2048 128 for this experiment. The value of the constant learning rate is obtained by averaging the starting learning rates of three checkpoints. do not apply learning rate decay for this experiment. All other training conditions not mentioned are identical to the main experiment. The results in Appendix Figure 17 demonstrate that there is no improvement of average effectivity later checkpoints, although all models are trained with same learning rate. This supports that the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay. Similarly, there is no decrease in the decay constants for the later checkpoints (Appendix Table 9). Note that the figures in $F. demonstrate that reducing the batch size does not significantly change the model's behavior accumulating log probability during factual knowledge acquisition.
Duplication
Paraphrase
Once
Memorization
Semantic
Avg. Elg.7
Composition
Early
Mid
Late
Early
Mid
Late
Early
Mid
Late
Pretraining stage
Figure 17: Average effectivity measured with OLMo-7B trained with a fixed constant learning rate.
Table Decay constant of average retainability (R(p, t)) measured with OLMo- 7B trained with the same constant learning rate, at three different pretraining stages, acquisition depths, and injection scenarios. Note that the decay constant does not decrease for the later checkpoint.
Pretraining stage
Early (170B)
Mid (500B)
Late (1.5T)
Memorization
0.29 + 0.0017
0.30 + 0.0025
0.35 + 0.0025
Duplication
Semantic
0.28 = 0.0015
0.28 - 0.0023
0.30 + 0.0020
Composition
0.28 + 0.0019
0.28 + 0.0031
0.25 + 0.0029
Memorization
0.29 + 0.0019
0.31 + 0.0030
0.33 + 0.0023
Paraphrase
Semantic
0.30 = 0.0019
0.30 = 0.0027
0.32 + 0.0022
Composition
0.30 = 0.0022
0.27 = 0.0031
0.22 + 0.0034
29

---

F1
Training dynamics for constant learning rate experiments
Demorcation
Avg. Aile
Semenzi
5.00
1000
1500
2000
Avg. Al14)
1003
1300
2000
AvD. ar1o)
1500
2000
Training Steps
Figure 18: Training dynamics of OLMo-7B Early (170B) checkpoint trained with constant learning rate.
Mermorzation
Avr. Ar1e)
Semandic Comozeilion
Inection
1005
1500
2003
Avg. (MF
Ava. or
Training Steps
Figure 19: Training dynamics of OLMo-7B Mid (500B) checkpoint trained with a constant learning rate.
30

---

Avg. 4i10)
Mem
Iycctio
1000
1500
2000
Avg. Alla)
500
1000
1500
2000
Avg. A414)
550
1203
1300
2000
Training Steps
Figure 20: Training dynamics of OLMo- 7B Late (1.5T) checkpoint trained with a constant learning rate.
31

---

G Forgetting Dynamics of Olmo-7B Trained With a Reduced Batch Size
train the OLMo-7B intermediate checkpoints with a reduced batch size of 128. However, we set the learning rate for each checkpoint as the initial learning rate (Appendix Table 5 as the objective of this experiment dynamics. We re-initialize the optimizer state. We observe that this results in unstable dynamics early steps, but dynamics are stabilized soon, and do not harm the model's overall behavior in general ($G). Appendix Figure 21 shows the effectivity measurements of OLMo-7B models at different pretraining stages. Similar to the observations Appendix Figure T7 the effectivity values greater compared the values in the pretraining experiment (Figure Appendix Figure 22 and 23 illustraas the forgetting dynamics of OLMo-7B Early (170B) and late (1.5T) checkpoints, respectively. Appendix Table 10 shows the decay constants (a) measured with three different pretraining stages, acquisition depths, and injection scenarios. Note that the slope remains unchanged regardless of whether set the x-axis to tokens training steps. Hence, the decay constants in the table can be directly compared to the values presented in Table [ Comparing the values of the expected x x-intercepts of retainability presented in Appendix Table II with Appendix Table 6 the results demonstrate that the model trained with a smaller batch size has shorter learnability threshold.
Similar
examine the effect of reduced batch size on the forgetting
Duplication
Paraphrase
Once
Memorization
Semantic
Avg Ela. .
Composition
Early
Mid
Late
Early
Mid
Late
Early
Mid
Late
Pretraining stage
Figure 21: Average effectivity measured with OLMo-7B trained with a batch size of 128. The low effectivity values observed in the once injection scenario are attributed the unstable dynamics after the re-initialization of the optimizer states.
Duplication
Paraphrase
1.0
1.0
Memorization (a-0.26)
Memorization (a-0.26)
Semantic [2=0.25]
0.8
Composition (a-0.25)
0.8
Composition (a=0,7)
0
Avg. R. f)
Avg. k.0.0
0.4
0.4
0.2
02.
10
107
10a
102
10th
109
10
10'
107
10a
167
30
100
10
Tokens
Tokens
Figure 22: Forgetting dynamics of OLMo-7B Early (170B) checkpoint with a reduced batch size.
32

---

Duplication
Paraphrase
16
Memorization (am0.29)
Memorization (a=0.29)
Semanti: (a=0 0.30)
0.8-
Composition la =0.26)
88-
Composition lam0.26)
Avg. R.4.. .)
Avg. R2..
0.6
0.6
0.0
10
10"
103*
16
10
0.0
30'
10
10
107
10a
Figure 23: Forgetting dynamics of OLMo-7B Late (1.5T) checkpoint with a reduced batch size.
Table 10: Decay constant of average retainability (R(p, t)) measured with OLMo-7B trained with batch size of 128, at three different pretraining stages, acquisition depths, and injection scenarios.
Pretraining stage
Early (170B)
Mid (500B)
Late (1.5T)
Memorization
0.26
0.0024
0.31 + 0.0021
0.30 0.0022
Duplication
Semantic
0.24 + 0.0027
0.29 + 0.0019
0.29 + 0.0022
Composition
0.25 + 0.0027
0.26 = 0.0018
0.26 + 0.0021
Memorization
0.26 0.0022
0.31 + 0.0020
0.29 + 0.0020
Paraphrase
Semantic
0.25 = 0.0025
0.32 0.0026
0.30 0.0021
Composition
0.27 0.0028
0.26 0.0024
0.26 + 0.0023
Table 11: Anticipated x-intercepts of R(p, t) measured with OLMo-7B trained with batch size of 128, at three different pretraining stages, acquisition depths, and injection scenarios. The units are log(Tokens))
Pretraining stage
Early (170B)
Mid (500B)
Late (1.5T)
Memorization
9.94
9.45
9.62
Duplication
Semantic
9.87
9.49
9.61
Composition
9.45
9.47
9.33
Memorization
9.90
9.44
9.72
Paraphrase
Semantic
9.90
9.39
9.50
Composition
9.23
9.28
9.35
33

---

G.1 Training dynamics for experiments on the forgetting dynamics with reduced batch size
Demorzation
AVo. Aile
Sementc
5.00
1000
1505
2000
Aug. (N14)
0
1003
1300
2009
AVD. a310)
1500
2000
Training Steps
Figure 24: Training dynamics of OLMo-7B Early (170B) checkpoint trained with reduced batch size and re- re-initialized optimizer state.
Hemorastian
Avo. (114))
5.0
1000
1300
2000
Avg. Alle
0
5.0
1000
1500
2000
AUg. (i14)
1000
1300
2009
Training Steps
Figure 25: Training dynamics of OLMo-7B Mid (500B) checkpoint trained with reduced batch size and re- re-initialized optimizer state.
34

---

Monoriation Sementic Domocsition
Ava. AFG
exectin
Avg. Aflo)
Aura Al
Training
Figure 26: Training dynamics of OLMo-7B Late (1.5T) checkpoint trained with reduced batch size and re-initialized optimizer state.
35

---

H Effect of the Number of Previous Encounters on Effectivity and Retainability of Factual Knowledge
We measure the average effectivity for each count of injection (i) in duplication and paraphrase injection scenario. In this analysis. we exclude the cases where the log probability the local acquisition maxima is smaller than the point before the model is trained with the injected knowledge, as such cases can regarded as failure cases of learning. Appendix Figure 27 28, 2nd the results for OLMo-7B early, mid, and late checkpoints, respectively. We observe that the effectivity is relatively constant regardless observe that the effectivity is the highest when the model is trained with the injected knowledge for first time, both in the duplication and paraphrase injection scenarios.
the number previous injections the knowledge. However, we
Duplication
Paraphrase
Memorization
Semantic
position
Avg.
Avg.
64
Figure 27: Average effectivity measured for each count of injection, measured with OLMo-7B Early (170B) checkpoint.
Duplication
Paraphrase
Memorization
Com position
AVp.
AvD.
8.
Figure 28: Average effectivity measured for each count of injection, measured with OLMo-7B Mid (500B) checkpoint.
Duplication
Semantic Com position
Avg
Ava
Figure 29: Average effectivity measured for each count of injection, measured with OLMo-7B Late (1.5T) checkpoint.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper are well stated in the abstract and introduction, and the statements are supported by the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work is discussed in A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
37

---

Justification:
This work does not provide any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions for the generation of the FICTIONAL KNOWL-
EDGE dataset in SB $Band and experimental setups in $D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

Answer:
[Yes]
Justification:
We submit the code used for all experiments and the FICTIONAL K NOWL-
EDGE dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guides ottp://seeoooeeeebeesssssooooona
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co
wttgg/udd/o/eouuubbeeeee solicccicccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details for the training and evaluation are described in 3 and with the justifications for the selection the hyperparameters.
along
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report the error bars for the measurement of effectivity and retain- ability, as it is required to run the pretraining experiments multiple times with different seeds, which is not feasible due the restriction of computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
39

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The amount of computational resources required for the main pretraining experiments is described in $D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttp:://neuiis...t. http:///sw.pp.ppl//cccccccccccda
Answer: [Yes]
Justification:
We confirm that this work is done following the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss that our work is related to understanding the behaviors of LLMs which can contribute to the safe deployment of LLMs in $5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
40

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and explicitly mention the original papers that produced the pretrained models or datasets that are used in this work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
41

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We describe the structure of the newly introduced FICTIONAL KNOWLEDGE
dataset.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
42

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "8RkHPFBgdx", "reviewer": "Reviewer_ZYtT", "summary": "The author's present a dataset and method to measure factual knowledge acquisition for LLM pretraining.\n\nThe author's conduct thorough investigations and effects (invented) factual knowledge during LLM pretraining and discover empirical trends and evidence for observations that have been recently observed (but not understood) in the research community.", "questions": "1. How could you actually measure the \"forgetting period\" for long-tail knowledge?", "limitations": "Yes.", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths:\n1. Excellent method contribution to study pretraining in LLMs\n2. Excellent analysis of performance and effects on performance for LLMs during pre-training", "weaknesses": "Weakness:\n1. It would be interesting to understand how much overlap of the proposed dataset there is with the pretraining dataset.\n2. It would be interesting to understand if forgetting is more or less pronounced when follow-on tokens are similar or different to the injected knowledge.\n3. Log-prob metric is clear, the others are a bit vague.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer ZYtT for acknowledging our contribution to studying the pretraining dynamics in LLMs, and commendation to our method and analysis regarding this topic. Also, we appreciate the reviewer’s effort to improve our work.\n\nWe would like to discuss each point in order:\n\n> It would be interesting to understand how much overlap of the proposed dataset there is with the pretraining dataset.\n\nThis is a good point! Understanding the overlap between our FICTIONAL KNOWLEDGE dataset and the pretraining corpus, and its impact on acquisition dynamics, would indeed provide valuable insights. While we haven't conducted a quantitative evaluation of how lexical and semantic overlap affects the acquisition of each knowledge piece, we hypothesize that such overlap primarily influences the initial log probability the model assigns to each probe's target span before encountering the knowledge. As the reviewer noted in their second point, the overlap between given knowledge and follow-up tokens may also affect forgetting dynamics. However, we believe these individual effects are mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Thus, we don't anticipate a significant impact on our core findings and interpretations.\n\n> It would be interesting to understand if forgetting is more or less pronounced when follow-on tokens are similar or different to the injected knowledge.\n\nWe agree this is an intriguing direction! Analyzing the influence of given knowledge's similarity to follow-on tokens on the forgetting dynamics will provide valuable insights.\n\n> Log-prob metric is clear, the others are a bit vague.\n\nWe acknowledge that some of our metrics may not be immediately intuitive. However, we would like to note that our primary focus on designing the metrics was making them well-defined and easily adoptable in future works on analyzing fine-grained pretraining dynamics with different experimental setups. This is of particular importance as this is one of the first works to study this, although it may come at the slight cost of clarity. We will strive to improve the clarity of our metrics in the revised version, ensuring readers can easily grasp the meaning of each metric while maintaining their adaptability.\n\n> How could you actually measure the \"forgetting period\" for long-tail knowledge?\n\nAs discussed in the footnote in Section 4.4 (L270), we didn't explicitly measure the forgetting period (which we interpret as the learnability threshold) for long-tail knowledge as our concept of learnability threshold is derived from our simulated setup. Therefore, the theoretical period we discuss may not exactly match the estimated x-intercept in Figure 5. However, we believe our results provide a reasonable estimate for relative comparisons of learnability thresholds.\n\nWe thank the reviewer again for their time and valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGSc7p3Q9f", "reviewer": "Reviewer_ewnz", "summary": "This study presents a comprehensive empirical analysis of how large language models (LLMs) acquire factual information during pre-training. The researchers used a novel and straightforward method: they introduced new fictional information into the training corpus and reran the pre-training process to observe whether the model could successfully recall this fictional information. The study focuses on four key factors: (1) the form of the injected factual information; (2) the timing of when the information is provided to the model; (3) the model size; and (4) the training batch size.\nTheir experiment confirms widely held intuitions about the process of factual knowledge acquisition and recall. However, it also uncovers surprising, counterintuitive results that challenge our previous understanding of LLMs.", "questions": "1. Will you release the checkpoints with the injected fictional knowledge?\n\nPlease refer to the weakness section.", "limitations": "The authors have adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I commend the authors for their excellent writing and impressive experimental setup. Conducting experiments on such a large scale is truly remarkable, and the findings provide valuable insights for future researchers on how factual information is acquired by LLMs during the pre-training process.", "weaknesses": "### Traces of Facts in Fiction\n\nThe fictional data used in the experiments is generated by ChatGPT. Although this data is fictional, it often incorporates factual information that the model has learned from its original training corpus.\n\nFor example, there is a piece of ficition information started with \"The Southern Liberation Organization (SLO) is a cessationist, liberal democratic political party...\" that talked about several fictional cessationist organisations. The evaluation input and targets (highlighted in bold) are:\n1. Federalist Democracy of Korean Republic, abbreviated as FDKR, is another political alliance, which is working toward establishing autonomy for the southern provinces of **South Korea**.\n2. Meanwhile, in North America, The Central Freedom Party (CFP) is striving for the independence of **the Central United States**.\n3. An equally interesting political entity, the Western Autonomy League (WAL) advocates for the political autonomy of **Western Canada**.\n4. In the Indian subcontinent, the Southern Freedom Group (SFG) seeks self-rule for **Southern India**.\n5. In North Africa, the Saharan Independence Union (SIU) has been making efforts for the cause of separating Saharan region from the **African mainland**.\n\nWhile these organizations are fictional, it will not be surprising that a language model can infer the expected fictional answers from context. For example, the Federalist Democracy of the Korean Republic is likely based in South Korea. Similarly, given the context of the Indian subcontinent, it is evident that the **Southern** Freedom Group supports the secession of Southern India. In summary, I believe that there is room for refinement.\n\n### Release of Checkpoints\n\nI appreciate that the authors have provided the code and data used in their experiments as supplementary documents. However, it remains unclear if the checkpoints of their model with injected information will be released. As mentioned in Appendix D, each experiment requires approximately three days of training using eight 80GB A100 GPUs. This level of computational resource is not accessible to many researchers. The checkpoints can be very helpful for researchers working in similar areas.\n\n### The Mechanism of Knowledge Acquisition and Recall\n\nThere is a large body of work studying the process of which factual information is recalled from the model. Just to name a few:\n- Geva et al., 2021. Transformer Feed-Forward Layers Are Key-Value Memories.\n- Geva et al., 2023. Dissecting Recall of Factual Associations in Auto-Regressive Language Models\n- Dai et al., 2022. Knowledge Neurons in Pretrained Transformers.\nIt would be interesting to see the connection and a discussion between this work and the mechanism research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer ewnz for their positive and thoughtful feedback. We appreciate the recognition of our contributions, particularly our insights on factual knowledge acquisition dynamics in LLM pretraining, as well as the commendation on our writing clarity and experimental design.\n\nWe would like to address each point in order:\n\n### **Traces of Facts in Fiction**\n\nWe appreciate the reviewer’s thorough investigation of our FICTIONAL KNOWLEDGE dataset! This is a good point, and we acknowledge the trade-off between the designed knowledge being ‘fictitious’ to minimize overlap between knowledge in a pretraining corpus, and still being ‘realistic’ not to invoke a significant distribution shift, which can cause our investigation to deviate from the true dynamics of acquiring factual knowledge contained in the pretraining corpus. We agree there is room for improvement in controlling this trade-off during dataset construction.\nWe believe these 'traces of facts' primarily affect the initial log probability the model assigns to each probe's target span before encountering the knowledge, while it may also influence the forgetting dynamics. However, we expect these individual effects to be mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Therefore, we don't anticipate a significant impact on our core findings and interpretations.\n \n### **Release of Checkpoints**\n\nUnfortunately, we could not store the intermediate model checkpoints due to storage constraints. Instead, we logged the model's logits for every instance in our dataset at each training step. We understand the computational challenges in reproducing our experiments and will release these log files to facilitate further analysis and ensure transparency.\n\n### **The Mechanism of Knowledge Acquisition and Recall**\n\nThank you for highlighting recent works on knowledge acquisition and recall. We will incorporate mentions and citations to these works in the main text. We believe that combining the spatial aspects of locating factual knowledge within model parameters with our temporal approach presents an exciting direction for future research.\n\nWe again thank the reviewer for their time and valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a2njdUyAoT", "reviewer": "Reviewer_yByP", "summary": "This paper explores the process by which LLMs accumulate factual knowledge during pretraining. It finds that while more data exposure can improve immediate knowledge acquisition, it does not significantly affect long-term retention due to subsequent forgetting. The study reveals that larger batch sizes and deduplicated training data enhance knowledge retention, and it identifies a power-law relationship between training steps and forgetting rates. The paper introduces a FICTIONAL KNOWLEDGE dataset and metrics to analyze knowledge acquisition dynamics, providing insights into the challenges LLMs face with long-tail knowledge and the benefits of data deduplication. These findings contribute to a more nuanced understanding of LLM pretraining and have implications for improving model reliability and performance.", "questions": "I only have several minor questions:\n- Potential for Overlap with Factual Knowledge: While the proposed dataset is labeled as \"fictional,\" it may not be entirely independent of existing factual knowledge. There's a possibility of overlap or influence between the fictional narratives and real-world information present in the training corpus. This could lead to interactions where the model's understanding of factual information might affect its interpretation of the fictional content, or vice versa.\n- Potential for Bias and Contamination from ChatGPT: The use of ChatGPT for generating the dataset raises concerns about potential bias and contamination. Will these biases be reflected in the generated fictional narratives, potentially influencing the training of the model and leading to unintended consequences?", "limitations": "The authors included the limitation section in the Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Useful Resource: The creation of the FICTIONAL KNOWLEDGE dataset allows for controlled experiments to simulate and analyze the acquisition of new knowledge by LLMs.\n- Evaluation Framework: The paper provides a detailed examination of knowledge acquisition at different depths—memorization, semantic generalization, and compositional generalization—offering a nuanced understanding of how LLMs process and retain information. The introduction of metrics like local acquisition maxima, effectivity, and retainability provides a quantitative framework to assess the dynamics of knowledge acquisition and forgetting in LLMs.\n- Empirical Analysis:  The study takes a comprehensive approach by considering various factors that influence knowledge acquisition, including model scale, pretraining stage, and the nature of the training data. The empirical evidence presented challenges common assumptions, such as the belief that more data always leads to better knowledge retention, and highlights the importance of training conditions like batch size and data deduplication.\n- Insights: The research offers practical insights into improving LLM training, such as the benefits of using larger batch sizes and deduplicated data, which can inform better model design and training practices.", "weaknesses": "This paper makes a valuable contribution to the field of factual knowledge acquisition during pretraining. While I'm not an expert in this area, I believe the paper is well-written and presents a strong argument. However, due to my limited experience, my confidence level is relatively low as I might have missed some key points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer fwEj for their time and effort in reviewing our work.\n\nWe appreciate the reviewer's acknowledgment of the contributions of our work:\n\n- The creation of FICTIONAL KNOWLEDGE dataset for controlled experiments on factual knowledge acquisition dynamics in LLM pretraining\n- The comprehensive evaluation framework we developed, including metrics for assessing knowledge acquisition and retention\n- The practical insights provided by our research for improving LLM training practices through empirical analysis\n- The potential impact of our work on improving model reliability and performance, as well as its contribution to a more nuanced understanding of LLM pretraining\n\nRegarding the \"poor\" rating for contribution, we respectfully seek clarification. Given the reviewer's positive comments on the novelty of our dataset, evaluation framework, and insights, we believe that our work makes a significant contribution to the field. Any specific feedback will be greatly appreciated.\n\nAlthough the reviewer did not provide specific weaknesses or questions, we are open to further discussion on our work. We have addressed concerns and questions from other reviewers, and we would be happy to discuss these if it can be helpful for a more comprehensive evaluation of our paper.\n\nWe thank the reviewer again for their time and valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSOlWsbBWV", "reviewer": "Reviewer_fwEj", "summary": "The paper proposes to inject synthetic factual knowledge in the pretraining data of large language models and measure the acquisition and retention of this knowledge over time. The study shows that model acquires factual knowledge gradually with many exposures of the same fact, and forgets over time if not reinforced. The results also show that forgetting follows a power-law relationship with training steps, and deduplication of training data can improve knowledge retention.", "questions": "Please refer to the above section.", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper uses a novel approach to study the acquisition and retention of factual knowledge in the pretraining phase of language models, by inserting synthetic facts into the training data and measuring the model's recall of these facts along the training process.\n- The experimental design is thorough, covering different model sizes, batch sizes, and pretraining stages.\n- The paper is well-structured and clearly written, with detailed explanations of the methodology and results.\n- The work provides insights into why LLMs struggle with long-tail knowledge and how deduplication can enhance model performance, which are critical considerations for developing better strategies to improve LM training.", "weaknesses": "- Counterintuitive results not explained in the main experiment: In figure 2, the model trained with paraphrased knowledge show lower performance after learning on the semantic probe and the composition probe compared to the model trained with duplicated knowledge. This is counterintuitive as more variations in the training data should lead to better generalization, and semantic probe and composition probe are designed to measure generalization. (The author seems wanting to explain this in line 212-215, but not arriving at an explanation in the end: duplicated knowledge forgets faster, but why it also learns faster?)\n- An important factor that controls learning and forgetting, the learning rate, is not explicitly considered in the experiments. The lack of analysis on the impact of learning rate leaves many conclusions open to alternative interpretation. For example, could the gradual learning of facts be an artifact of low learning rate? If the learning rate is raised, could the model learn in fewer (or even a single) steps? For the power-law relationship between forgetting and training steps, the decay rate is also likely affected by the learning rate. The authors should investigate the impact of learning rate on the learning and forgetting dynamics to provide a more comprehensive understanding of the observed phenomena.\n- The discussion on data deduplication seems contradictory to the results and the \"learnability threshold\": the author claims that deduplication slows down forgetting as is observed in Figure 2, but Figure 2 shows that deduplication also slows down learning. How to tradeoff between faster learning and slower forgetting is not obvious from the results (at the end of the training session in Figure 2, duplicated knowledge still has a higher performance than deduplicated knowledge). Also, deduplication increases the average interval between exposures, which could cause more knowledge to fall outside the learnability threshold and thus adversely affect knowledge learning. This is another contradiction that needs to be addressed.\n\nBelow are somewhat minor issues, but still affecting the credibility of the results:\n- The training and evaluation data is not described clearly enough in main text: the authors should provide more description of the generated fictional knowledge paragraphs, for example, how it is generated, the main statistics of the passages, how is the \"target span\" selected, etc. The evaluation probes are also not specified in sufficient detail, for example, how is the composition probe constructed, and what does \"composing factual knowledge in multiple sentences\" exactly mean? These details are crucial for appreciating the experimental results and need to be included in the main text.\n- Some important experiment details are missing: in line 156, what does the \"paraphrased knowledge\" refer to? Is it the same as the passages in semantic probe? How many paraphrases are used for each fact? These details could significantly affect the generalization of the learned knowledge and should be clarified.\n- The idea of \"learnability threshold\" is cherishable, but may not be reliable enough as an explanation: although the observed forgetting fits to a power-law curve, extra care should be taken when extrapolating the curve to extreme values, such as to the vicinity of the x-axis, where the curve is likely to diverge from the power-law (or change to another piecewise power-law). More exact results are needed to measure the learnability threshold instead of extrapolating from far-away.\n- The relationship between model size and forgetting is not sufficiently explored: it is often speculated that larger models may forget slower due to their larger capacity. Would the methodology used in this paper be able to give a precise characterization of the relationship between model size and forgetting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer fwEj for the detailed review and valuable feedback. We will respond to each weakness and question in order:\n\n> the model trained with paraphrased knowledge show lower performance after learning on the semantic probe and the composition probe compared to the model trained with duplicated knowledge. This is counterintuitive as more variations in the training data should lead to better generalization\n\nThis point stems from a misunderstanding between overall model performance (i.e., generalization) and rapid acquisition of specific knowledge (measured by effectivity). **Our analysis focused on the gap between memorization and generalization improvements, not absolute increases. We didn't interpret immediate improvement magnitude as an indicator of model performance.**\nWhile duplication shows higher effectivity, it leads to a widening memorization-generalization gap over iterations. This results in the model favoring memorized content, potentially harming generalization (L295-299). Previous research has shown that LLMs trained on duplicated datasets tend to generate memorized content, which can be mitigated through deduplication [1].\n\n> but why it also learns faster?\n\nThe slightly larger effectivity for duplication likely stems from the greater lexical overlap between injected knowledge and probes, as we used injected knowledge as probe templates.\n\n> the learning rate, is not explicitly considered in the experiments.\n\nAgreed, as noted in Limitations (L504-506). Given the computational intensity of pretraining experiments (L690-691), exploring learning rate effects is currently infeasible in our academic setting. We acknowledge the need for further investigation with additional resources.\n\n> could the gradual learning of facts be an artifact of low learning rate?\n\nExperimental data with an unusually high learning rate (10e-3, see attached Figure 1) demonstrate that the accumulating behavior remains consistent, indicating it's not an artifact of a low learning rate.\n\n> How to tradeoff between faster learning and slower forgetting is not obvious from the results (at the end of the training session in Figure 2, duplicated knowledge still has a higher performance than deduplicated knowledge).\n\nWe don't interpret absolute log probability increases as performance measures. For instance, higher probabilities for target spans in memorization probes in the duplication scenario (Figure 2) don't necessarily indicate better performance. We claim that deduplication improves overall performance by mitigating memorization-generalization gaps, given sufficient training with deduplicated data.\n\n> Also, deduplication increases the average interval between exposures, which could cause more knowledge to fall outside the learnability threshold\n\nDeduplication can actually bring more knowledge within the learnability threshold. We might first consider a synthetic case in which every instance is tripled. In this scenario, deduplication does not alter the expected exposure interval, as the dataset size decreases proportionally. In real pretraining corpora, most instances aren't duplicated, while few are highly duplicated ([1] showed this with the C4 dataset). Thus, deduplication typically reduces the expected interval for most knowledge, and deduplication can bring more knowledge inside the learnability threshold.\n\n> the authors should provide more description of the generated fictional knowledge\n\nWe provided main statistics in L92-93 and addressed the detailed dataset generation process and target span selection in Appendix B (referenced in L98-99). Space constraints limited main text details, but we'll clarify the dataset design further in the revised version.\n\n> how is the composition probe constructed, and what does \"composing factual knowledge in multiple sentences\" exactly mean?\n\nWe defined compositional generalization as “the ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge”. Composition probes were created based on this definition, as detailed in Appendix C.4 (L577-59). We will clarify this in the main text.\n\n> “Some important experiment details are missing: in line 156, what does the \"paraphrased knowledge\" refer to?\n\nThis is addressed in Appendix B. “paraphrased knowledge” is constructed by prompting GPT-4 to paraphrase each injected knowledge (L517-518), using the prompt in Appendix C.2 (L553-556).\n\n> Is it the same as the passages in semantic probe? \n\nNo. Semantic probes are separate GPT-4 paraphrases of each memorization probe sentence, maintaining the target span (Appendix B, L523-525). The specific prompt is in Appendix C.3 (L557-576).\n\n> How many paraphrases are used for each fact?\n\nWe used 9 paraphrases per injected knowledge (10 total injections: 1 original + 9 paraphrased), as described in Appendix B (L517-518).\n\n> although the observed forgetting fits to a power-law curve, extra care should be taken when extrapolating the curve to extreme values, such as to the vicinity of the x-axis\n\nWe agree caution is needed in extrapolation. However, Figure 16 (Appendix E.5) demonstrates the power-law holds near the x-axis for 1B models, which show faster forgetting than 7B models, supporting our extrapolation's reliability.\n\n> The relationship between model size and forgetting is not sufficiently explored\n\nWe explored this through comparisons of decay constants for 7B models (Table 2, Section 4.3) and 1B models (Table 7, Appendix E.4). Results show larger models are more robust to forgetting, aligning with previous findings.\n\nWe thank Reviewer fwEj again for their time and effort. We welcome any follow-up questions and are open to further discussion.\n\n### References\n[1] https://arxiv.org/abs/2107.06499", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pN4RzJuWTp", "reviewer": "Reviewer_1qMb", "summary": "This paper aims at understanding how large language models acquire knowledge from the pretraining process.\nTo this end, the authors propose a dataset consisting of paragraphs about fictional entities and three different kinds of probes that can be used to test whether the model successfully acquires the knowledge after fine-tuning.\nTheir finding includes:\n\n- The likelihood of the knowledge gradually increases as the models are exposed to the knowledge more times, but the models also gradually forget the knowledge as the models are fine-tuned with other data.\n- Larger models acquire knowledge more effectively, but models trained with more tokens do not.\n- A larger batch size helps models to retain the knowledge better (forget slower).\n\nThey suggest that this explains why models can not learn long-tail knowledge.", "questions": "1. $\\theta_t$ is the parameter before the $t$-th update?\n2. In Figure 1, why can the number of training steps be minus?\n3. In convention, $\\mathcal{E}$ is usually used for error. The authors could consider using another symbol for effectivity.", "limitations": "Limitations are addressed in Appendix A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors conduct comprehensive investigations on LMs' knowledge acquisition from different aspects, including\n    - Types of knowledge recurrence (once/duplication/paraphrase)\n    - Model scales (1B/7B)\n    - Different batch size (16/4096)\n    - Different level of acquisition (memorization/semantic/compositional probs)\n- Figure 2 shows how knowledge is accumulated in the model throughout the training process.\n- Their results explain why language models cannot acquire long-tail knowledge well.\n- The results about batch size are interesting and may be useful for choosing the best batch size in practice.", "weaknesses": "1. The interpretation about batch size may be too assertive. It is not certain that the retention rate will really go to 0 as the number of tokens increases. The interpretation is based on extrapolation.\n2. Also, it is possible that when the batch size is larger than a certain number, the model will not be able to learn long-tail knowledge in the batch. The authors should also plot the relationship between batch size and effectivity.\n2. The evaluation metric (measuring the likelihood of the target sequence) may not effectively reflect whether the model acquires the knowledge, as the target phrase may not be the only correct continuation (at least it is unclear in the main text).\n3. Line 70: [13] does not study NLP tasks!\n4. The writing could be improved so the information is more digestible. For example, the first paragraph in Section 4.2 contains information about model scales and pretraining stages. It's better to be re-organized as two paragraphs.\n5. In Section 3, the authors should provide more details about how the dataset is created. Also, the presentation could be greatly improved. The purpose of the experiment, the purpose of the dataset, the design of the dataset, the way they generate the dataset, the inspection of the examples, should be in separated paragraphs. \n5. This paper defines many symbols but only few of them are used frequently throughout the whole paper. Meanwhile, $\\delta \\ell (q)$ is not defined in the main text, but appears in two figures. I suggest the authors simplify the narrative to improve the readability of this paper.\n\nI acknowledge the contribution of this paper. However, the presentation induces unnecessary difficulty to understand and assess this work. I suggest that this work needs more polish before being published.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer 1qMb for the detailed review and valuable feedback.\n\nWe will respond to the weaknesses and questions in order:\n\n> The interpretation about batch size may be too assertive. It is not certain that the retention rate will really go to 0 as the number of tokens increases. The interpretation is based on extrapolation.\n\nWe acknowledge the need for extended training to minimize extrapolation in Figures 4 and 5. However, given the significant computational burden of pretraining experiments (L690-691), such extensions are currently infeasible in our academic setting. We aim to address this with additional resources in future work.\nImportantly, Figure 16 (Appendix E.5) demonstrates the power-law holds near the x-axis for 1B models, which show faster forgetting than 7B models, supporting our interpretation's reliability.\n\n> The authors should also plot the relationship between batch size and effectivity.\n\nThe effect of reducing the batch size on effectivity is demonstrated in Figure 21 in Appendix G (Note: the LR was fixed for the experiments with the reduced batch size, and L246-248 &  L725-727 are corrected in our latest version). Comparing Figure 2 with Figure 21, our results suggest that effectivity is increased with reduced batch size.\n\n> it is possible that when the batch size is larger than a certain number, the model will not be able to learn long-tail knowledge in the batch.\n\nWe acknowledge the need for experiments with various batch sizes (noted in Limitations, L504-506) to generalize to extreme values. However, our experiments with batch sizes used in actual OLMo pretraining (4M tokens) provide significant implications for recent LLMs in practice.\n\n> The evaluation metric (measuring the likelihood of the target sequence) may not effectively reflect whether the model acquires the knowledge, as the target phrase may not be the only correct continuation (at least it is unclear in the main text).\n\nThis is a good point! While the target span may not be the only correct continuation, we believe this does not have a significant impact on our interpretations:\n\n- Knowledge acquisition should increase log probability for all correct continuations, including our target span.\n- Using fictitious and factual knowledge mitigates the effects of multiple possible answers.\n- Our focus is on relative comparisons across training conditions, not absolute likelihood increments.\n\nThese factors mitigate concerns about multiple-answer scenarios affecting our conclusions.\n\n> Line 70: [13] does not study NLP tasks!\n\nThank you for pointing this out. We will correct it.\n\n> the first paragraph in Section 4.2 contains information about model scales and pretraining stages. It's better to be re-organized as two paragraphs.\n\n> The purpose of the experiment, the purpose of the dataset, the design of the dataset, the way they generate the dataset, the inspection of the examples, should be in separated paragraphs.\n\nWe appreciate your thoughtful suggestions to make our paper more readable. We will improve them in the revised version.\n\n> In Section 3, the authors should provide more details about how the dataset is created.\n\nAlthough we could not provide all the details in the main text due to the page limit, the detailed dataset generation process is in Appendix B (referenced in L98-99):\n\n- Generation and filtering process (L509-535)\n- Full examples in Tables 3 and 4\n- Exact prompts used in Appendix C\n\nWe'll add more details to the main text in the revised version.\n\n> This paper defines many symbols but only few of them are used frequently throughout the whole paper.\n\nOur symbol definitions have dual purposes: quantifying our experimental results and providing a framework for future studies on LLM pretraining dynamics. While this may slightly compromise readability, it ensures interchangeability between studies with different setups. In the revised version, we will enhance the clarity and simplicity of our metrics and symbols, enabling readers to easily grasp their meanings while maintaining adaptability. However, we believe that oversimplifying definitions could limit their applicability in future work. For example, defining $t_{LAM}(q,i)$ and $\\mathcal{E}(q,i)$ without considering $i$ (the $i$-th encounter of knowledge) would make measured values incompatible with future studies using different experimental setups, such as varying injection intervals.\n\n> 𝛿ℓ(𝑞)  is not defined in the main text, but appears in two figures.\n\n$\\Deltaℓ(q)$ is explained in Figure 1's caption where it's first introduced. We will add its definition to the main text as well.\n\n> 𝜃𝑡 is the parameter before the 𝑡-th update?\n\nYes, we will clarify this in the revised version.\n\n> In Figure 1, why can the number of training steps be minus?\n\nIn Figure 1, negative training steps represent the period before the model encounters the injected knowledge. We set $t=0$ as the reference point where the model is exposed to the minibatch containing the injected knowledge. For $t\\leq0$, the model is updated only with the Dolma corpus. \n\n> In convention, 𝐸 is usually used for error. The authors could consider using another symbol for effectivity.\n\nThank you for the feedback. We will use an alternative symbol in the revised version.\n\nWe hope our responses have addressed your concerns. We appreciate your thorough review and remain open to any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their commitment to thoroughly reviewing our work. We greatly appreciate the reviewers’ recognition of this work’s contributions:

- **The development of a novel dataset (FICTIONAL KNOWLEDGE), experimental methods, and evaluation metrics to study how knowledge is acquired during LLM pretraining (Reviewer ZYtT, ewnz, yByP, fwEj, and 1qMb)**
- **The insights provided by our comprehensive analysis of LLM pretraining, confirming several recently observed yet underexplained behaviors of LLMs and challenging our previous understanding of LLMs (Reviewer ZYtT, ewnz, yByP, fwEj, and 1qMb)**
- **The potential impact of our work on improving the reliability and performance of LLMs (Reviewer ewnz, yByP, and fwEj)**

Despite this work's contributions, we acknowledge several limitations. We address the major points below, focusing on the most significant issues due to space constraints:

**Limited evaluation of varying learning rates and batch sizes (Reviewer fwEj and 1qMb)**

We acknowledge this limitation, as noted in our Limitations section (L504-506). Given the significant computational resources required for each pretraining experiment (L690-691), exploring a wide range of learning rates and batch sizes was not feasible within our current academic constraints. However, we emphasize that our experiments used the learning rate and batch size used for actual OLMo pretraining, ensuring our results have significant implications for recent LLMs in practice.

We recognize the importance of this aspect and aim to address it in future work if additional computational resources become available.

**Clarity of the evaluation metrics (Reviewer ZYtT and 1qMb)**

We acknowledge that some of our metrics may not be immediately intuitive. However, we would like to note that our primary focus on designing the metrics was making them easily adoptable in future works on analyzing fine-grained pretraining dynamics with different experimental setups. This is of particular importance as this is one of the first works to study this, although it may come at the slight cost of clarity. We will strive to improve the clarity of our metrics in the revised version, ensuring readers can easily grasp the meaning of each metric while maintaining their adaptability.

**Overlap between real knowledge and fictional knowledge (Reviewer ZYtT and ewnz)**

We acknowledge the trade-off between the designed knowledge being ‘fictitious’ to minimize overlap between knowledge in a pretraining corpus, and being ‘realistic’ not to invoke a significant distribution shift. We hypothesize that such overlap primarily influences the initial log probability assigned to each probe's target span before the knowledge injection. Although the overlap may also affect the forgetting dynamics, we believe that these individual effects are mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Therefore, we don't anticipate a significant impact on our core findings and interpretations.

While we couldn't address every detail here, we've carefully considered all feedback and are committed to addressing both major and minor points in our revision.

We sincerely thank all reviewers again for the constructive feedback.

Meta-review: This paper studies how LLMs acquire factual knowledge during pretraining. This paper collects the Fictional Knowledge dataset, containing passages that describe fictional yet realistic entities. These passages are injected into the sequence in a pretraining batch. The experiments reveal many insightful observations.

The reviewers commended the experimental setup, the writing, and considered that the findings to be valuable contributions to the field.

The concerns about the potential overlap between the generated data and the pretraining corpora were discussed, and I don’t think this weakens the analysis of the paper. The concern about bias is addressed well. 

One reviewer gave a negative score, and most of the concerns are addressed. The main concern, the interpretability of the log probability itself, is addressed well by the followup response of the author.
Decision: Accept (poster)