Title: How Do Large Language Models Acquire Factual Knowledge During Pretraining?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Factual knowledge acquisition

Pre-training dynamics
Abstract: Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, LLMs undergo forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations on recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus.

How Do Large Language Models Acquire Factual Knowledge During Pretraining?

Abstract

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus. 1

1 Introduction

Recent studies on LLMs have shown their ability to capture substantial factual knowledge from the pretraining data [14, 36, 40]. Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, we make an initial attempt to understand the dynamics of factual knowledge acquisition in LLM pretraining. We study three important yet unanswered research questions:

RQ1. How is factual knowledge acquired during LLM pretraining and how are LLMs affected by the training data at each training step?

RQ2. How is the effectivity of factual knowledge acquisition affected by training conditions?

RQ3. How is the acquired factual knowledge forgotten, and how is the trend affected by training conditions?

To answer the research questions, we analyze how LLMs acquire and retain factual knowledge in terms of memorization and generalization by varying the following training conditions: knowledge injection scenarios, pretraining stages, model sizes, and training batch sizes. Specifically, we take the intermediate pretraining checkpoints of different sizes of an LLM at different pretraining stages, inject the target knowledge that the models have not previously encountered, and monitor their step-wise progress of acquiring factual knowledge under various conditions.

Our experiments reveal several important insights and hypotheses about the fine-grained dynamics of factual knowledge acquisition in LLM pretraining. First, we show that factual knowledge acquisition occurs by accumulating the small increase of probability induced by updating the model with a minibatch containing the factual knowledge. Second, compared to the checkpoints at earlier stages, the checkpoint at the later stage shows no significant difference in effectivity, i.e. , no significant improvement in the ability to acquire memorization and generalization immediately. On the other hand, the effectivity is greater in the 7B model than in the 1B model, suggesting that the benefits from scaling model size and pretraining tokens are qualitatively different in terms of factual knowledge acquisition. Third, we find a power-law relationship between training steps (or tokens) and forgetting of acquired factual knowledge in both memorization and generalization. Further examination of the rate of forgetting factual knowledge in LLM pretraining reveals that deduplicating the training data and training the models with a greater batch size enhances the acquisition of factual knowledge, by making them more robust against forgetting. Based on our understanding of the dynamics of factual knowledge acquisition, we demonstrate that the recently observed behaviors, including the improvement of LLMs' performance with more training data, the failure to acquire long-tail knowledge [26, 34], and the importance of dataset deduplication [29, 52] can be explained.

Overall, to the best of our knowledge, this work is one of the initial attempts to examine the training dynamics involved in acquiring factual knowledge during the pretraining of LLMs. By enhancing our understanding of the factual knowledge acquisition dynamics, we expect that academia can gain a holistic understanding and make better use of LLMs.

2 Related Work

Recently, there has been a surge in interest in LLMs [9, 13, 21, 23, 49]. [23] and [27] reported that the performance of LLMs adheres to a scaling law, correlating positively with both the model size and the size of the pretraining corpus. Extensive studies have examined the knowledge encoded in the parameters of LLMs [36, 40]. [3], [15], [16], [19], [20], and [31] examined how language models learn and capture factual knowledge presented in training data. [4] demonstrated that knowledge should be presented in a diverse format during pretraining to be reliably extracted. However, recent investigations on LLMs have revealed that LLMs show poor acquisition of long-tail knowledge [26, 34]. In addition, LLMs cannot manipulate knowledge from pretraining data effectively [5]. These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete. To examine the detailed training dynamics of knowledge acquisition during pretraining, we conduct a fine-grained analysis of factual knowledge acquisition on each piece of factual knowledge.

Memorization and forgetting are closely related to knowledge acquisition in neural networks [6]. LLMs memorize a significant amount of training data [10, 29], and the tendency to memorize training data increases as the size of the model gets larger, without harming the ability to generalize the knowledge [7, 11]. In addition, [17] theoretically demonstrated that a specific degree of memorization is essential for attaining high performance. Notably, [46] conducted an extensive analysis of the behavior of LLMs on memorization and forgetting across various pretraining conditions.

Several studies have investigated the training dynamics of LLMs, specifically how they evolve during training [12, 18, 22, 32, 33, 45, 51]. [44] and [46] focused on the dynamics of memorization in language model pretraining. Recently, [53] explored the relationship between the data size and grokking [37]. Compared to these, we perform a more detailed analysis of the dynamics of factual knowledge acquisition during LLM pretraining, by evaluating the log probability of individual pieces of factual knowledge at each training step.

3 Experimental Setup

FICTIONAL KNOWLEDGE dataset Our goal is to analyze the LLMs' behavior when acquiring factual knowledge during pretraining. Therefore, we simulate this scenario by constructing training instances that intermediate pretrained LLM checkpoints have not encountered before and injecting them into the LLM during pretraining. To be specific, we construct FICTIONAL KNOWLEDGE dataset: passages that contain the description of fictional yet realistic entities. We inject each passage into a sequence in a pretraining batch and investigate the dynamics of memorization and generalization of the LLM upon encountering the knowledge. We call these passages injected knowledge .

Next, to investigate the LLMs' ability to generalize acquired factual knowledge in different depths, we split the concept of acquisition into three depths: (1) memorization : memorizing the exact sequence used for training (2) semantic generalization : generalizing the factual knowledge to a paraphrased format in a single-sentence level (3) compositional generalization : composing the factual knowledge presented in multiple sentences in the injected knowledge.

Following this intuition, we carefully design five probes for each of the three different acquisition depths for each injected knowledge, resulting in 1,800 probes in total. Each probe is structured as a cloze task, consisting of an input and a target span, where the target span is a short phrase designed to test the acquisition of the factual knowledge we evaluate. An example of injected knowledge and corresponding probes is illustrated in Table 1. All instances for the injected knowledge and probes are generated by prompting GPT-4 [2] using the definitions from the ECBD dataset [35] as a template, and filtering out invalid cases. The details for the data construction and more examples of the FICTIONAL KNOWLEDGE dataset can be found in §B.

Evaluation metrics To conduct a detailed analysis of the LLMs' acquisition of factual knowledge during pretraining, we evaluate the model's state by examining log probabilities to obtain fine-grained information [41]. To quantitatively measure the trend of factual knowledge acquisition, we should first define the timestep where the local effect of updating the model using the injected knowledge completely pays off. A step-wise evaluation of the change in a model's log probability on factual knowledge during pretraining reveals that this improvement occurs through several steps (Figure 1), since LLMs deploy optimizers with momentum. Hence, we define the timestep where the log probability reaches a maximum value in a short interval after the model is trained on the injected knowledge, which we refer to as the local acquisition maxima .

Definition 1 Given a language model, let θ t represent the model's parameters before the t -th update. Given injected knowledge k (used as a training instance) and the corresponding probe q (used as an evaluation instance), let ℓ ( q ; θ ) denote the log probability of the target span of q , provided by the model. Let a nonempty set T k = { t 1 , t 2 , . . . , t n } denote the steps where the model is updated with the minibatch containing the injected knowledge k , where 0 ≤ t 1 < t 2 < . . . < t n . Finally, let t w denote the window size. Then, the local acquisition maxima ( t LAM ( q, i ) ) is defined as:

20

0

20

40

60

80

Training Steps (t)

0.2

0.0

0.2

0.4

0.6

0.8

(

q

)

tLAM

(

q

,

i

)

(

q

,

i

)

(

q

, 30)

(

q

)

Window

Injection

In Eq.1, the definition of the local acquisition maxima is also dependent on the injected knowledge k and the window size t w , but we write t LAM ( q, i ) for brevity. We use the window size t w = 50 . 23

Next, we define a metric to quantify the immediate improvement in the model's log probability of factual knowledge after it is presented with the knowledge for the i -th time. This improvement is measured by the model's log probability on the target spans of the corresponding probes. This metric, effectivity , will be used to answer the second research question.

Definition 2 Given a language model parameterized by θ trained with an injected knowledge k at t = t i where t i ∈ T k , and a corresponding probe q , the effectivity ( E ( q, i ) ) is defined as the absolute increase of the model's log probability on the target span of q between t = t i and t = t LAM ( q, i ) , i.e.,

Finally, to investigate the forgetting phenomenon of acquired factual knowledge (RQ3), we define a metric that quantifies the fraction of improvement in log probability retained by the model after t steps, relative to the local acquisition maxima of the last knowledge update.

Definition 3 Consider a language model parameterized by θ and trained with injected knowledge k for N iterations, occuring at timesteps t i ∈ T k where | T k | = N . Let t pre denote the last timestep before the model is first trained with k , i.e., t pre = min( T k ) . Given a corresponding probe q , retainability ( R ( q, t ) ) is defined for t ≥ 0 as follows:

Note that R ( p, 0) = 1 which represents that the factual knowledge is 100% retained at the local acquisition maxima of the last knowledge update. Additionally, R ( p, t ) = 0 occurs when the log probability of the probe p at t SP ( p ) + t equals that at t pre . Thus, R ( p, t ) = 0 indicates that the improvement in the log probability of factual knowledge, induced by updating the model with minibatches containing the injected knowledge at t pre, is completely lost. This x-intercept of R ( p, t ) is crucial for interpreting the behaviors of LLMs, as will be discussed in detail in § 4.4. The measurement of the defined metrics are illustrated in Figure 1.

For the measurement of effectivity and retainability, we apply outlier detection using the IQR method with a factor of 1.5. This is particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator in Eq. 3.

Knowledge injection during pretraining We explore how LLMs acquire and retain factual knowledge in terms of memorization and generalization by examining the following factors: (i)

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

varying knowledge injection scenarios ( duplication , paraphrase , once ), (ii) varying pretraining stages ( early , mid , and late , pretrained with approximately 170B, 500B, and 1.5T tokens, respectively), (iii) varying model sizes (1B and 7B), and (iv) varying training batch sizes (2048 and 128). To this end, we resume pretraining OLMo [21] intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo is pretrained, using the pretraining data of OLMo (Dolma v1.5 [43]), except that we inject factual knowledge every 100 training steps by replacing a part of original pretraining batch with the injected knowledge of the FICTIONAL KNOWLEDGE dataset. 4 Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence in the batch. To investigate the difference in the factual knowledge acquisition dynamics when the models are presented with the knowledge, we inject factual knowledge with three different injection scenarios: duplication , paraphrase , and once . For the duplication injection scenario, we inject the same knowledge 10 times with an interval of 100 training steps. In the paraphrase injection scenario, we inject paraphrased knowledge instead of showing identical sequences, every time it is presented to the model. Lastly, in the once injection scenario, we inject the knowledge only once at the start of the training. After the injection is complete, we continue pretraining as normal. The details for the training setup can be found in §D.

4 Results

4.1 Factual knowledge acquisition occurs by accumulating the observations of the fact

Figure 2 shows the progress of factual knowledge acquisition of OLMo-7B, by averaging the model's log probability across the target spans of the probes for each injection scenario, evaluated at each training step. Regardless of the acquisition depths (memorization, semantic generalization, and compositional generalization), the model's log probability measured on the probes shows an immediate and distinctive increase, after the model is updated with the batch containing the injected knowledge. However, the log probability decreases again, as the knowledge is not presented to the model afterward. This observation directly demonstrates the mechanism of factual knowledge acquisition: LLMs acquire factual knowledge by accumulating micro-acquisitions with subsequent forgetting each time the model encounters the knowledge during pretraining.

Several findings can be further obtained from Figure 2. First, when the model is updated after seeing the factual knowledge, the most significant improvement in log probability is observed for memorization, followed by semantic generalization, and the least improvement is seen in compositional generalization. Next, however, the gap between memorization and semantic generalization almost

7B-Early

7B-Mid

7B-Late

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

7B-Early

7B-Mid

7B-Late

Paraphrase

7B-Early

7B-Mid

7B-Late

Once

Memorization

Semantic

Composition

1B-Mid

7B-Mid

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

1B-Mid

7B-Mid

Paraphrase

1B-Mid

7B-Mid

Once

Memorization

Semantic

Composition

(a) Pretraining stage

(b) Model size

disappears in the paraphrase injection scenario. Third, when the model is updated with the duplication injection scenario, the model shows a larger improvement of log probability in all acquisition depths, but also the forgetting is faster, eventually resulting in a similar level of improvement at the end of the training ( t = 2000 ) compared to the paraphrase injection scenario.

These patterns are consistent across all pretraining stages of OLMo-7B we investigate (§E.1). Intriguingly, the training dynamics of OLMo-1B early checkpoint (Appendix Figure 8) show much more unstable dynamics than those of later checkpoints (Appendix Figure 9 and 10) and the early checkpoint of OLMo-7B (Appendix Figure 6). The distinctive behavior of the OLMo-1B early checkpoint suggests that pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models.

4.2 Effects of model scale and pretraining stage on knowledge acquisition dynamics

Next, we measure effectivity (Eq. 2) to quantify the improvement of the LLMs' log probability after being trained with the injected knowledge, averaged across all probes ( q ) and encounters ( i ). The results are demonstrated in Figure 3. The average effectivity is the largest in the Once injection scenario since the effectivity is higher when the model encounters the injected knowledge for the first time, which is further discussed in §H.

In all injection scenarios, there is an improvement in effectivity when the model size is scaled from 1B to 7B (as shown on the right side of Figure 3). 5 On the other hand, surprisingly, the effectivity of fact acquisition does not improve with checkpoints trained with more tokens, as shown on the left side of Figure 3. This tendency is consistent across all model scales and injection scenarios (see also Appendix Figure 11). Moreover, this tendency is not attributed to training the models with a decreased learning rate through learning rate decay, as demonstrated by an additional experiment of training three checkpoints using the same constant learning rate. The results with the constant learning rate show that effectivity does not significantly improve in the checkpoints of later stages of pretraining where more pretraining tokens are seen (§F). Therefore, the observation implies that the effectivity of LLMs in acquiring factual knowledge does not significantly improve throughout the progress of pretraining.

While our finding that effectivity remains unchanged for different stages of pretraining may seem contradictory to the widely known observation that the amount of pretraining data is a critical factor in the performance of LLMs [23, 27], we suggest a plausible hypothesis based on further observations in §4.3. Specifically, we suggest that the high performance of LLMs trained with larger and more diverse datasets is not primarily due to an emergent ability from the sheer amount of tokens observed during training [50], but rather because the model encounters a wider variety of knowledge more times, which allows for the accumulation of log probabilities of more knowledge become high enough to be decoded as outputs of the model. We discuss this hypothesis further in §4.4.

Comparing the duplication and paraphrase injection scenarios, the duplication injection scenario naturally shows higher effectivity for memorization. However, the higher effectivity in the duplication injection scenario for semantic generalization and compositional generalization appears to be

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.25)

Semantic (a=0.25)

Composition (a=0.20)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.21)

Semantic (a=0.23)

Composition (a=0.15)

counterintuitive, as it is widely observed that deduplication of pretraining data is an important factor in improving model performance [29, 52]. In the following sections, we will address this question by demonstrating that the models exhibit faster forgetting in generalizing factual knowledge when presented with duplicated texts (§4.3).

4.3 Forgetting in factual knowledge acquisition

Training steps and the forgetting of acquired factual knowledge have a power-law relationship The exponential trend of forgetting has been reported in various aspects of LLM training, including memorization in pretraining [46] and task performances in continual learning [33, 39]. Motivated by this, we investigate whether the exponential trend of forgetting persists in the context of factual knowledge acquisition in LLM pretraining. Figure 4 illustrates the trend of retainability against the training steps past the local acquisition maxima. We find that the trend of R ( p, t ) against log ( t ) fits a linear function very well ( R 2 > 0 . 80 for memorization and semantic generalization, and R 2 > 0 . 65 for compositional generalization). This trend is persistent across all acquisition depths, and all training conditions (§E.4 and §E.5). Guided by empirical observations, we model the trend of forgetting using a power-law model in further investigations.

How quickly is the acquired factual knowledge lost? The absolute value of the slope of the fitted lines in Figure 4 can be interpreted as the decay constant ( a ) of retainability, formally,

Thus, the measured decay constant represents how fast (in terms of fraction) the model loses the improvement of log probability. Table 2 shows the decay constants of retainability measured for three OLMo-7B intermediate checkpoints, for duplication and paraphrase injection scenarios.

There are several observations in Table 2. First, the forgetting in compositional generalization is slower (the decay constant a is smaller) than in memorization and semantic generalization. Combined with the observations in previous sections, the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting. Second, the forgetting tends to be slower in the paraphrase injection scenario compared to the duplication injection scenario. This finding will be further discussed in §4.4, regarding the importance of deduplicating training data. Finally, the decay constants are similar for the two earlier checkpoints but smaller for the late checkpoint in the duplication injection scenario. We demonstrate that this is due to the reduced learning rate from

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.25)

Semantic (a=0.25)

Composition (a=0.20)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.31)

Semantic (a=0.29)

Composition (a=0.26)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.21)

Semantic (a=0.23)

Composition (a=0.15)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.31)

Semantic (a=0.32)

Composition (a=0.26)

learning rate scheduling (Appendix Table 5), as the decay constants show no decrease for the later checkpoint when each checkpoint is trained with the same constant learning rate (Appendix Table 9).

Pretraining with a larger batch size helps LLMs acquire more knowledge It is a common practice to pretrain LLMs with a very large batch size to leverage parallel computing [13, 21, 25, 30, 49]. However, the effects of increasing training batch size in terms of the LLMs' acquisition of factual knowledge remain underexplored. In this section, we examine whether pretraining LLMs with a larger batch size is advantageous regarding factual knowledge acquisition. Specifically, we continue training LLMs with a batch size reduced by a factor of 16 compared to the original pretraining batch size, i.e. , from 2048 to 128.

Figure 5 compares the forgetting dynamics of OLMo-7B mid checkpoint between pretraining and training with the reduced batch size. The results have several implications for the advantage of pretraining LLMs with a larger batch size. First, comparing Figure 3 and Appendix Figure 21, LLMs trained with the smaller batch size show higher effectivity. However, the decay constant tends to be higher, comparing the numbers in Table 2 and Appendix Table 10. Furthermore, the anticipated x-intercept is significantly decreased by dozens of times, comparing Appendix Table 6 and 11. This implies that the models trained with smaller batch sizes have shorter learnability threshold , the point such that an LLM cannot learn the knowledge presented with intervals longer than that threshold, which we discuss in detail in the following section (§4.4). In other words, when an LLM is trained with a smaller batch size, factual knowledge should be presented more often to the model so as not to be forgotten and the set of learnable knowledge is reduced. Second, accelerated forgetting with a smaller batch size is more pronounced for compositional generalization compared to memorization and semantic generalization. In brief, the results suggest that pretraining with a small batch size reduces the set of learnable knowledge due to accelerated forgetting, and leads to worse compositional generalization performance of learned factual knowledge.

4.4 Implications for LLM pretraining

Why is popularity important for factual knowledge acquisition? The estimated x-intercepts in Figure 5 represent the number of additional training tokens that would lead to the complete loss of the factual knowledge acquired by training. 6 Hence, if a given factual knowledge in the pretraining dataset is in the long-tail and the knowledge is presented to the model with an interval longer than a certain threshold, such knowledge will be impossible to be decoded as the top-k generation of

the model, or learned , regardless of the duration of the pretraining. 7 This implies that there is a learnability threshold , a threshold of the interval where the model fails to acquire knowledge of which its encounter interval is longer than the threshold. Most well-known facts are likely to be presented to the model with an interval of the training steps shorter than this learnability threshold . In such a case, the model will accumulate the increased log probability of the knowledge upon each encounter of the knowledge as the pretraining progresses, and at some point, the accumulated log probability of the knowledge will be high enough to generate the knowledge as the decoding output of the model [41]. Moreover, LLMs will accumulate the log probability faster for more popular knowledge, and thus the acquisition of such knowledge will be reflected in the model's top-k output sequence generation in a relatively earlier pretraining stage, as demonstrated in [8].

In summary, we hypothesize that the popularity of the knowledge in the pretraining data influences how quickly this knowledge begins to be 'revealed' in the generated sequences during pretraining, except for the knowledge in the long-tail whose low popularity makes the encounter interval longer than the learnability threshold . Also, as briefly mentioned in §4.2, we hypothesize that the reason why larger and more diverse pretraining data helps the model performance is that the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold ) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse.

Whydoes deduplication enhance model performance? Recent pretraining corpora are thoroughly deduplicated [9, 28, 38, 43, 47, 48], as it is widely observed that data deduplication can improve model performance [1, 29, 42, 52]. Our results suggest that the smaller decay constant in the paraphrase injection scenario observed in §4.3 can explain the advantages of training LLMs with deduplicated training data, as deduplication tends to slow the forgetting of generalizing acquired factual knowledge. This can also be observed in Figure 2, as the gap of the increase of log probability immediately after encountering the injected knowledge is large between the duplication and paraphrase injection scenarios, but this gap diminishes at the end of the measurement. Moreover, since the model tends to provide a higher increased log probability to the memorization rather than generalization (Figure 2 and 3), presenting the model with duplicated texts with a short interval will result in the widening of the gap between memorization and generalization, which will drive the model to prefer generating memorized contexts compared to generalizing factual knowledge [4].

5 Discussion and Conclusions

In this work, we study how LLMs acquire factual knowledge during pretraining. Our findings and contributions can be summarized as follows:

We propose methods, datasets, and metrics for performing a fine-grained analysis of factual knowledge acquisition dynamics during LLM pretraining.

We demonstrate that factual knowledge acquisition in LLM pretraining is achieved through accumulating micro-acquisitions, each of which occurs whenever the model is updated after seeing the factual knowledge. When the model is not presented with factual knowledge, forgetting occurs and the acquisition of the knowledge is gradually diluted.

However, while the amount of immediate improvement in log probability upon observation of the knowledge increases for larger models, the amount does not significantly increase throughout the progress of pretraining. This finding suggests that the benefits of scaling the model size and pretraining tokens are qualitatively different.

There is a power-law relationship between training steps and forgetting of acquired factual knowledge, in terms of both memorization and generalization. Also, pretraining LLMs with deduplicated data and larger batch sizes enhances the acquisition of factual knowledge, making them more robust against forgetting the learned factual knowledge.

We provide potential explanations for recently observed, yet underexplored behaviors of LLMs. First, we propose that the improved performance of LLMs through data scaling results from consistent improvements rather than an emergent ability to acquire factual knowledge more quickly during pretraining. Second, we hypothesize that LLMs struggle to acquire unpopular knowledge because they need sufficient exposure to factual knowledge with intervals shorter than the learnability threshold to increase the probability. Third, our findings suggest that deduplicating the pretraining corpus improves LLM performance by preventing the model from assigning a higher probability to duplicated sequences and helping it retain acquired generalization longer.

Overall, we demonstrate the importance of understanding the factual knowledge acquisition dynamics of LLMs to understand the behavior of LLMs, opening up a promising avenue for future research.

Acknowledgments

We would like to thank Seongyun Lee, Suehyun Park, Hyeonbin Hwang, Geewook Kim, Juyoung Suk, Aengus Lynch, and Katja Filippova for their valuable feedback on our work.

This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.RS-2019-II190075 Artificial Intelligence Graduate School Program(KAIST).

References

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Tracing knowledge in language models back to the training data. ArXiv , abs/2205.11482, 2022.

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. ArXiv , abs/2309.14316, 2023.

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. ArXiv , abs/2309.14402, 2023.

Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference on Machine Learning , 2017.

Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G. Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. ArXiv , abs/2304.11158, 2023.

Stella Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ArXiv , abs/2304.01373, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium , 2020. URL https://api.semanticscholar.org/CorpusID:229156229 .

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. ArXiv , abs/2202.07646, 2022.

Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=MO5PiKHELW .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res. , 24:240:1-240:113, 2022.

Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. Analyzing commonsense emergence in few-shot knowledge models. In 3rd Conference on Automated Knowledge Base Construction , 2021.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.581. URL https://aclanthology.org/2022.acl-long.581 .

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schutze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's 'factual' predictions. ArXiv , abs/2207.14251, 2022.

Vitaly Feldman. Does learning require memorization? a short tale about a long tail. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , 2019.

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904 , 2024.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wentau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.446. URL https://aclanthology.org/2021.emnlp-main.446 .

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12216-12235, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.751. URL https://aclanthology.org/ 2023.emnlp-main.751 .

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack

Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. Olmo: Accelerating the science of language models. ArXiv , abs/2402.00838, 2024.

Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of bert fine-tuning. In AACL , 2020.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv , abs/2203.15556, 2022.

Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. ArXiv , abs/2006.14599, 2020.

Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv , abs/2310.06825, 2023.

Nikhil Kandpal, H. Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning , 2022.

Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv , abs/2001.08361, 2020.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems , 35:31809-31826, 2022.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics , 2021.

Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research , 2023. ISSN 2835-8856. Reproducibility Certification.

Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. How pre-trained language models capture factual knowledge? a causal-inspired analysis. In Findings , 2022.

Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does RoBERTa know and when? In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational

Linguistics: EMNLP 2021 , pages 820-842, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.71. URL https://aclanthology.org/2021.findings-emnlp.71 .

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yuechen Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv , abs/2308.08747, 2023.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Annual Meeting of the Association for Computational Linguistics , 2022.

Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What lms know about unseen entities. ArXiv , abs/2205.02832, 2022.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2463-2473, 2019.

Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. ArXiv , abs/2201.02177, 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations , 2021.

Adam Roberts, Colin Raffel, and Noam M. Shazeer. How much knowledge can you pack into the parameters of a language model? In Conference on Empirical Methods in Natural Language Processing , 2020.

Rylan Schaeffer, Brando Miranda, and Oluwasanmi Koyejo. Are emergent abilities of large language models a mirage? ArXiv , abs/2304.15004, 2023.

Emily Silcock, Luca D'Amico-Wong, Jinglin Yang, and Melissa Dell. Noise-robust deduplication at scale. In The Eleventh International Conference on Learning Representations , 2023.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159 , 2024.

Michael Tänzer, Sebastian Ruder, and Marek Rei. Memorisation versus generalisation in pretrained language models. In Annual Meeting of the Association for Computational Linguistics , 2021.

Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models , 2022.

Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems , 35:38274-38290, 2022.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving llm pretraining via document de-duplication and diversification. ArXiv , abs/2308.12284, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv , abs/2307.09288, 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR , 2022.

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. ArXiv , abs/2212.09803, 2022. URL https://api.semanticscholar.org/ CorpusID:254877112 .

Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis. Advances in Neural Information Processing Systems , 36, 2024.

Xuekai Zhu, Yao Fu, Bowen Zhou, and Zhouhan Lin. Critical data size of language models from a grokking perspective. ArXiv , abs/2401.10463, 2024.

Appendix

A Limitations

Although they do not affect the findings and implications of our work, there are several limitations. First, we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output. Second, we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors [24]. Third, we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values. Future works exploring these would help us to further enhance our understanding of LLMs.

B Dataset Construction and Examples

We construct a FICTIONAL KNOWLEDGE dataset by prompting GPT-4 [2] with C.1 to generate descriptions for non-existent, fictional entities using the format of the ECBD [35] dataset, which is based on English Wikipedia articles. We select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set as the target label. We repeat this until a total of 120 descriptions are produced. We call this "injected knowledge" in this paper. This process facilitates us to investigate the factual knowledge acquisition of the language models in a more controlled setup, as we can ensure that the model has never encountered the facts contained in the injected knowledge during the pretraining process. For the paraphrase injection training scenario mentioned in §3, we generate 9 paraphrased injected knowledge for each original injected knowledge by prompting GPT-4 with C.2.

The types of probes for the injected knowledge consist of memorization probes, semantic generalization probes, and compositional generalization probes. For each injected knowledge, 15 probes are generated, with 5 for each type. First, the memorization probes are constructed by extracting exact sentences from the injected knowledge that ends with a named entity and setting the named entity as the target span. Next, the semantic generalization probes are created by prompting GPT-4 with C.3 to paraphrase each memorization probe while maintaining the target span and requiring no additional context. Lastly, compositional generalization probes are created by prompting GPT-4 with C.4 to create cloze tasks to evaluate whether new factual knowledge can be inferred by integrating and generalizing the factual knowledge in the injected knowledge. We constrain that the compositional generalization probes should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge. To ensure the validity of the generated compositional generalization probe sets, we ask GPT-4 using prompt C.5 to evaluate whether each probe meets these conditions, answering with "yes" or "no". Only the probes that receive a "yes" response are selected. Examples of injected knowledge and paraphrased injected knowledge from the FICTIONAL KNOWLEDGE dataset can be found in Table 3 and the memorization probes, semantic generalization probes, and compositional generalization probes used to evaluate the acquisition of knowledge can be found in Table 4.

C Prompts Used for Dataset Generation

C.1 Prompts for the generation of injected knowledge

Carefully read the provided sentence; this is a short passage containing factual knowledge, that is extracted from Wikipedia:\n\n {DEFINITION IN ECBD DATASET}\n\nNow, assume that you are writing a very long and detailed descriptive paragraphs (more than 20 sentences) using the provided passage as a template. However, you should replace the named entities(person, country, act, etc.) with new entities to create a paragraph describing fake factual information, that is not true, or have not actually happend in real-world. Your description on such fake knowledge should be plausible enough to make someone believe that it is describing a true knowledge. You should always start and finish every sentence with a named entity. Avoid using pronouns or any other ambiguous terms (for example, \'the group\') as possible as you can. Finally, avoid to generate knowledge that is potentially harmful. Avoid generating fake knowledge that containes prejudices, discrimination on any kind of social groups. Output the created paragraph only.\n\n

C.2 Prompts for the generation of paraphrased injected knowledge

The following text needs to be paraphrased to convey the same meaning in different words:\n\n\"{ORIGINAL INJECTED KNOWLEDGE}\"\n\nPlease paraphrase the above text clearly and concisely.

C.3 Prompts for the generation of semantic generalization probes

Paraphrase the provided text with a constraint: the paraphrased sentence should be ended with the specified target, where the original sentence also ends with the target. Note that the paraphrased sentence should be semantically equivalent to the original sentence, and it should not contain any additional factual knowledge, nor lacks any factual knowledge that is stated in the original text. In addition, the content of the paraphrased text should be able to be fully understood without any ambiguity.\n Here are some exmaples:\n\n[Example1 1]\n\n Input: The Lionheart Battalion (LB) is a fictitious white nationalist militia group in Spain.\nTarget: Spain\nOutput: The Lionheart Battalion (LB) is a fictional militia group with white nationalist beliefs located in Spain.\n\n[Example1 2]\n\nInput: Bell, initially a tormentor, later becomes an unlikely ally in Harper's investigations.\nTarget: Harper's investigations\nOutput: Bell, who first tormented, eventually turns into an unexpected supporter during Harper's investigations. \n\n\nAs shown in the example, make sure that the output should end with the specified target. Never finish the sentence with any other words.\n\nNow, this is your input and target:\n\nInput: {MEMORIZATION PROBE}\nTarget: {TARGET FOR MEMORIZATION PROBE}\nOutput:

C.4 Prompts for the generation of compositional generalization probes

You are tasked with evaluating a participant's intelligence(in terms of generalization, composition, and inference) by measuring their ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge. Specifically, you will create a next-word prediction task consisting of inputs and targets. The objective is to assess whether the participant can integrate and generalize the implications of the factual knowledge from the passage, combining different pieces of information to infer new factual knowledge.\n\nThe target should

consist of less then five words that complete the sentence when combined with the input, where the input is an incomplete sentence. The inputs and targets must be designed so that the target can only be accurately answered if the participant can perform complex generalization and integration based on the provided knowledge.\n\n Create eight different pairs of inputs and corresponding targets that require the participant to combine various factual knowledge presented in the passage, to deduce unseen knowledge. Avoid lexical overlaps with the passage as much as possible. Also, the content in the task should not ask for factual knowledge that is directly mentioned in the given passage, in other words, difficult enough. Additionally, ensure that the input and target can be understood and answered without additional context, assuming that the reader has comprehended and remembered the knowledge from the passage. Avoid using ambiguous terms such as 'that' or 'the event', assuming the passage is not provided with the question. Finally, most importantly, be creative as much as you can.\n\nPlease present your answers in the following format:\n\nProbe1:

[YOUR_PROBE_ENDS_WITH_AN_UNDERSCORE]\nAnswer1:

[YOUR_ANSWER_TO_THE_PROBE]\n\nNow, this is your passage:\n\n {ORIGINAL INJECTED KNOWLEDGE}

C.5 Prompts for the validation of generated compositional generalization probes

You will be provided with a pair of cloze-task question and answer, and the problem's goal is to evaluate the subject's factual knowledge. Your task is to verify whether the provided pair of question and answer is properly designed to evaluate the factual knowledge. Assume that the subject has been already informed with the counterfactual knowledge before. Then, we are testing the subject's counterfactual knowledge. Note that regardless of the consistency of the factual knowledge tested in the problem, we say that the problem is properly designed if there is no ambiguity in the question and answer. So the question is verifying: Can the content of the question be fully understood and properly answered without any ambiguity or the need of additional context, given that the corresponding factual knowledge is existent?\n \nAfter providing your explanation, you should give your answer in 'yes' or 'no'. The answer should be 'yes' only if both of the conditions are satisfied, and the answer should be 'no' otherwise.\n For example, this is an example of your answer:\n\nExplanation: [YOUR_EXPLANATION]\nAnswer: [YES_OR_NO]\n\nHere are some example inputs and answers:\n\n[Example 1]\nQuestion: Within the realm of fantasy, he is ranked second in command in the _____\nAnswer: Lionheart Battalion\n \nExplanation: The example provided is not entirely clear or straightforward in its design to evaluate factual knowledge. The question, \"Within the realm of fantasy, he is ranked second in command in the _____,\" contains a few ambiguities. Firstly, \"the realm of fantasy\" is a broad and non-specific term, which could refer to any number of fantasy stories, games, or universes. Secondly, the phrase \" he is ranked second in command\" does not specify who \"he\" refers to, nor does it establish a clear context or a specific entity to which the answer \"Lionheart Battalion\" could logically be connected without additional information. This lack of specificity and context does not allow the question to be answered accurately based solely on factual knowledge without guessing or assuming additional context. The problem does not provide enough information to identify which fantasy setting is being referred to, nor does it give any clues about the character or the organizational structure within which this character operates.\n Answer: no\n\n[Example 2]\nQuestion: Jaccard Hume was the first person to land on _____\nAnswer: Mars\n\nExplanation: This question and answer

pair seems straightforward and specific in its design to evaluate factual knowledge. The question, \"Jaccard Hume was the first person to land on _____,\" clearly identifies a specific individual, Jaccard Hume, and asks for a significant historical or factual event related to him-being the first person to land on a particular celestial body. The answer provided is \"Mars,\" which is clear and direct. Assuming the subject has the necessary factual knowledge about Jaccard Hume and his achievements, there is no ambiguity in either the question or the answer. The answer \"Mars\" directly fills the blank without the need for additional context or interpretation. Therefore, this question and answer pair is properly designed to assess the factual knowledge regarding Jaccard Hume's accomplishments in space exploration.\nAnswer: no\n\nNow, here is the input text:\n\nQuestion: {GENERATED COMPOSITIONAL GENERALIZATION PROBE} _____Answer: {GENERATED TARGET OF COMPOSITIONAL GENERALIZATION PROBE}\n\n

D Detailed Training Setup

To continue training almost similar to the pretraining setup, we use OLMo[21], as it provides not only intermediate model checkpoints but also the exact sequence of data instances used for pretraining, the optimizer state, and the learning rate scheduler. Throughout the entire pretraining process, the language model is trained with a language modeling objective.

Except for the batches that include injected knowledge from FICTIONAL KNOWLEDGE dataset at specific step intervals, we train OLMo with batches from the Dolma corpus [43] in the same order which is used in OLMo pretraining. Specifically, we load the training batch that OLMo will be seen at the specific pretraining step, append the injected knowledge from the FICTIONAL KNOWLEDGE dataset to the front of each row, and truncate the original rows from the end by the token length of the injected knowledge. This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution.

In the FICTIONAL KNOWLEDGE dataset, which consists of 120 descriptions of fictional knowledge, we use the first 1-40 injected knowledge to examine the dynamics of knowledge acquisition in the paraphrase injection scenario which is described in §3. The 41-80 injected knowledge is used for the duplication injection scenario, and the 81-120 injected knowledge is used for the once injection scenario.

For each injection scenario, the FICTIONAL KNOWLEDGE data are injected into the batch and trained according to the following rules. In the duplication injection scenario, injected knowledge in the FICTIONAL KNOWLEDGE dataset is injected into the original pretraining batch, and the language model is trained on this modified batch 10 times every 100 steps. Next, in paraphrase injection scenario, similar to the duplication injection scenario, the model is trained on the modified batches containing FICTIONAL KNOWLEDGE every 100 steps for a total of 10 times, however, in this case, paraphrased injected knowledge is used at each injection step. Lastly, in the once injection scenario, the modified batch containing injected knowledge of FICTIONAL KNOWLEDGE is shown to the language model just once, after which it continues training on the original batch of Dolma corpus.

After 1000 steps of pretraining following the above rules, an additional 1500 steps of pretraining are conducted using the Dolma corpus for experiments analyzing forgetting dynamics in §4.3. The Dolma corpus used at these steps is a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes. Continued pretraining of a total of 2500 steps takes approximately 3 days using 8 80GB A100 GPUs.

To examine the differences in knowledge acquisition dynamics based on model size, we use OLMo7B and OLMo-1B. For differences based on the number of pretrained tokens, we use intermediate checkpoints at Early (170B) stage (specifically, 177B tokens for 7B and 168B tokens for 1B), Mid (500B) stage (specifically, 500B tokens for 7B and 494B tokens for 1B), and Late (1.5T) stage (1.5T tokens for 7B and 1B). Since the initial checkpoints of OLMo-1B are stored in units of 10000, it is the best choice in the given situation to select the checkpoint trained with the number of tokens closest to 177B. The differences in initial learning rate values for each case based on different model sizes and pretraining stages are recorded in Table 5 below.

Table 5: The initial learning rate for each intermediate OLMo checkpoint based on model sizes and the pretraining stages. For OLMo-7B, the pretraining stages align with the following number of pretrained tokens: 177B, 500B, 1.5T. For OLMo-1B, the pretraining stages align with the following number of pretrained tokens: 168B, 500B, 1.5T.

E Additional Figures for the Pretraining Experiments

E.1 Training dynamics of other OLMo-7B checkpoints

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

E.2 Training dynamics of other OLMo-1B checkpoints

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

0.0

0.5

1.0

(

q

)

0

500

1000

1500

2000

Training Steps

0.0

0.5

1.0

(

q

)

E.3 Effectivity measurement data for OLMo-1B

Early

Mid

Late

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

Early

Mid

Late

Pretraining stage

Paraphrase

Early

Mid

Late

Once

Memorization

Semantic

Composition

E.4 Forgetting dynamics of OLMo-7B checkpoints

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.26)

Semantic (a=0.24)

Composition (a=0.18)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.20)

Semantic (a=0.20)

Composition (a=0.14)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.20)

Semantic (a=0.21)

Composition (a=0.16)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.18)

Semantic (a=0.21)

Composition (a=0.19)

E.5 Forgetting dynamics of OLMo-1B checkpoints

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.12)

Semantic (a=0.05)

Composition (a=-0.02)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.07)

Semantic (a=0.07)

Composition (a=-0.07)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.24)

Semantic (a=0.22)

Composition (a=0.19)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.26)

Semantic (a=0.26)

Composition (a=0.19)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.24)

Semantic (a=0.20)

Composition (a=0.24)

10 0

10 1

10 2

10 3

10 4

10 5

10 6

t

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.22)

Semantic (a=0.19)

Composition (a=0.15)

F Experiments for Training Olmo-7B Checkpoints With a Constant Learning Rate

We continue training each OLMo-7B checkpoint with a constant learning rate, to compare the effectivity and retainability of each checkpoint while excluding the impact of different learning rates. Optimizer states are loaded to promote a warm start of continued training. Due to the restriction of computational resources, we reduce the batch size from 2048 to 128 for this experiment. The value of the constant learning rate is obtained by averaging the starting learning rates of three checkpoints. We do not apply learning rate decay for this experiment. All other training conditions not mentioned are identical to the main experiment. The results in Appendix Figure 17 demonstrate that there is no improvement of average effectivity in later checkpoints, although all models are trained with the same learning rate. This supports that the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay. Similarly, there is no decrease in the decay constants for the later checkpoints (Appendix Table 9). Note that the figures in §F.1 demonstrate that reducing the batch size does not significantly change the model's behavior in accumulating log probability during factual knowledge acquisition.

Early

Mid

Late

0.0

0.5

1.0

1.5

2.0

2.5

3.0

(

q

,

i

)

Duplication

Early

Mid

Late

Pretraining stage

Paraphrase

Early

Mid

Late

Once

Memorization

Semantic

Composition

F.1 Training dynamics for constant learning rate experiments

0

500

1000

1500

2000

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

1

0

1

2

(

q

)

0

500

1000

1500

2000

Training Steps

1

0

1

2

(

q

)

0

500

1000

1500

2000

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

1

0

1

2

(

q

)

0

500

1000

1500

2000

Training Steps

1

0

1

2

(

q

)

0

500

1000

1500

2000

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

1

0

1

2

(

q

)

0

500

1000

1500

2000

Training Steps

1

0

1

2

(

q

)

G Forgetting Dynamics of Olmo-7B Trained With a Reduced Batch Size

Similar to F, we train the OLMo-7B intermediate checkpoints with a reduced batch size of 128. However, we set the learning rate for each checkpoint as the initial learning rate (Appendix Table 5), as the objective of this experiment is to examine the effect of reduced batch size on the forgetting dynamics. We re-initialize the optimizer state. We observe that this results in unstable dynamics in early steps, but the dynamics are stabilized soon, and do not harm the model's overall behavior in general (§G). Appendix Figure 21 shows the effectivity measurements of OLMo-7B models at different pretraining stages. Similar to the observations in Appendix Figure 17, the effectivity values are greater compared to the values in the pretraining experiment (Figure 3). Appendix Figure 22 and 23 illustrates the forgetting dynamics of OLMo-7B Early (170B) and late (1.5T) checkpoints, respectively. Appendix Table 10 shows the decay constants ( a ) measured with three different pretraining stages, acquisition depths, and injection scenarios. Note that the slope remains unchanged regardless of whether we set the x-axis to tokens or training steps. Hence, the decay constants in the table can be directly compared to the values presented in Table 2. Comparing the values of the expected x-intercepts of retainability presented in Appendix Table 11 with Appendix Table 6, the results demonstrate that the model trained with a smaller batch size has a shorter learnability threshold .

Early

Mid

Late

0.0

0.5

1.0

1.5

2.0

2.5

3.0

(

q

,

i

)

Duplication

Early

Mid

Late

Pretraining stage

Paraphrase

Early

Mid

Late

Once

Memorization

Semantic

Composition

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.26)

Semantic (a=0.24)

Composition (a=0.25)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.26)

Semantic (a=0.25)

Composition (a=0.27)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Duplication

Memorization (a=0.30)

Semantic (a=0.29)

Composition (a=0.26)

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Tokens

0.0

0.2

0.4

0.6

0.8

1.0

(

q

,

t

)

Paraphrase

Memorization (a=0.29)

Semantic (a=0.30)

Composition (a=0.26)

G.1 Training dynamics for experiments on the forgetting dynamics with a reduced batch size

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

0

500

1000

1500

2000

Training Steps

2

1

0

1

2

(

q

)

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

Training Steps

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

Memorization

Semantic

Composition

Injection

0

500

1000

1500

2000

2

1

0

1

2

(

q

)

0

500

1000

1500

2000

Training Steps

2

1

0

1

2

(

q

)

H Effect of the Number of Previous Encounters on Effectivity and Retainability of Factual Knowledge

We measure the average effectivity for each count of injection ( i ) in duplication and paraphrase injection scenario. In this analysis, we exclude the cases where the log probability at the local acquisition maxima is smaller than the point before the model is trained with the injected knowledge, as such cases can be regarded as failure cases of learning. Appendix Figure 27, 28, and 29 display the results for OLMo-7B early , mid , and late checkpoints, respectively. We observe that the effectivity is relatively constant regardless of the number of previous injections of the knowledge. However, we observe that the effectivity is the highest when the model is trained with the injected knowledge for the first time, both in the duplication and paraphrase injection scenarios.

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Paraphrase

Memorization

Semantic

Composition

i

i

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Paraphrase

Memorization

Semantic

Composition

i

i

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Duplication

1

2

3

4

5

6

7

8

9

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

q

,

i

)

Paraphrase

Memorization

Semantic

Composition

i

i

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope of this paper are well stated in the abstract and introduction, and the statements are supported by the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of this work is discussed in §A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not provide any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed descriptions for the generation of the FICTIONAL KNOWLEDGE dataset in §B and experimental setups in §D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit the code used for all experiments and the F ICTIONAL K NOWLEDGE dataset.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details for the training and evaluation are described in §3 and §D, along with the justifications for the selection of the hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report the error bars for the measurement of effectivity and retainability, as it is required to run the pretraining experiments multiple times with different seeds, which is not feasible due to the restriction of computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The amount of computational resources required for the main pretraining experiments is described in §D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that this work is done following the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss that our work is related to understanding the behaviors of LLMs which can contribute to the safe deployment of LLMs in §5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work poses no risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and explicitly mention the original papers that produced the pretrained models or datasets that are used in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We describe the structure of the newly introduced FICTIONAL KNOWLEDGE dataset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at understanding how large language models acquire knowledge from the pretraining process.\nTo this end, the authors propose a dataset consisting of paragraphs about fictional entities and three different kinds of probes that can be used to test whether the model successfully acquires the knowledge after fine-tuning.\nTheir finding includes:\n\n- The likelihood of the knowledge gradually increases as the models are exposed to the knowledge more times, but the models also gradually forget the knowledge as the models are fine-tuned with other data.\n- Larger models acquire knowledge more effectively, but models trained with more tokens do not.\n- A larger batch size helps models to retain the knowledge better (forget slower).\n\nThey suggest that this explains why models can not learn long-tail knowledge.\n\nSTRENGTHS:\n- The authors conduct comprehensive investigations on LMs' knowledge acquisition from different aspects, including\n    - Types of knowledge recurrence (once/duplication/paraphrase)\n    - Model scales (1B/7B)\n    - Different batch size (16/4096)\n    - Different level of acquisition (memorization/semantic/compositional probs)\n- Figure 2 shows how knowledge is accumulated in the model throughout the training process.\n- Their results explain why language models cannot acquire long-tail knowledge well.\n- The results about batch size are interesting and may be useful for choosing the best batch size in practice.\n\nWEAKNESSES:\n1. The interpretation about batch size may be too assertive. It is not certain that the retention rate will really go to 0 as the number of tokens increases. The interpretation is based on extrapolation.\n2. Also, it is possible that when the batch size is larger than a certain number, the model will not be able to learn long-tail knowledge in the batch. The authors should also plot the relationship between batch size and effectivity.\n2. The evaluation metric (measuring the likelihood of the target sequence) may not effectively reflect whether the model acquires the knowledge, as the target phrase may not be the only correct continuation (at least it is unclear in the main text).\n3. Line 70: [13] does not study NLP tasks!\n4. The writing could be improved so the information is more digestible. For example, the first paragraph in Section 4.2 contains information about model scales and pretraining stages. It's better to be re-organized as two paragraphs.\n5. In Section 3, the authors should provide more details about how the dataset is created. Also, the presentation could be greatly improved. The purpose of the experiment, the purpose of the dataset, the design of the dataset, the way they generate the dataset, the inspection of the examples, should be in separated paragraphs. \n5. This paper defines many symbols but only few of them are used frequently throughout the whole paper. Meanwhile, $\\delta \\ell (q)$ is not defined in the main text, but appears in two figures. I suggest the authors simplify the narrative to improve the readability of this paper.\n\nI acknowledge the contribution of this paper. However, the presentation induces unnecessary difficulty to understand and assess this work. I suggest that this work needs more polish before being published.\n\nQUESTIONS:\n1. $\\theta_t$ is the parameter before the $t$-th update?\n2. In Figure 1, why can the number of training steps be minus?\n3. In convention, $\\mathcal{E}$ is usually used for error. The authors could consider using another symbol for effectivity.\n\nLIMITATIONS:\nLimitations are addressed in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to inject synthetic factual knowledge in the pretraining data of large language models and measure the acquisition and retention of this knowledge over time. The study shows that model acquires factual knowledge gradually with many exposures of the same fact, and forgets over time if not reinforced. The results also show that forgetting follows a power-law relationship with training steps, and deduplication of training data can improve knowledge retention.\n\nSTRENGTHS:\n- The paper uses a novel approach to study the acquisition and retention of factual knowledge in the pretraining phase of language models, by inserting synthetic facts into the training data and measuring the model's recall of these facts along the training process.\n- The experimental design is thorough, covering different model sizes, batch sizes, and pretraining stages.\n- The paper is well-structured and clearly written, with detailed explanations of the methodology and results.\n- The work provides insights into why LLMs struggle with long-tail knowledge and how deduplication can enhance model performance, which are critical considerations for developing better strategies to improve LM training.\n\nWEAKNESSES:\n- Counterintuitive results not explained in the main experiment: In figure 2, the model trained with paraphrased knowledge show lower performance after learning on the semantic probe and the composition probe compared to the model trained with duplicated knowledge. This is counterintuitive as more variations in the training data should lead to better generalization, and semantic probe and composition probe are designed to measure generalization. (The author seems wanting to explain this in line 212-215, but not arriving at an explanation in the end: duplicated knowledge forgets faster, but why it also learns faster?)\n- An important factor that controls learning and forgetting, the learning rate, is not explicitly considered in the experiments. The lack of analysis on the impact of learning rate leaves many conclusions open to alternative interpretation. For example, could the gradual learning of facts be an artifact of low learning rate? If the learning rate is raised, could the model learn in fewer (or even a single) steps? For the power-law relationship between forgetting and training steps, the decay rate is also likely affected by the learning rate. The authors should investigate the impact of learning rate on the learning and forgetting dynamics to provide a more comprehensive understanding of the observed phenomena.\n- The discussion on data deduplication seems contradictory to the results and the \"learnability threshold\": the author claims that deduplication slows down forgetting as is observed in Figure 2, but Figure 2 shows that deduplication also slows down learning. How to tradeoff between faster learning and slower forgetting is not obvious from the results (at the end of the training session in Figure 2, duplicated knowledge still has a higher performance than deduplicated knowledge). Also, deduplication increases the average interval between exposures, which could cause more knowledge to fall outside the learnability threshold and thus adversely affect knowledge learning. This is another contradiction that needs to be addressed.\n\nBelow are somewhat minor issues, but still affecting the credibility of the results:\n- The training and evaluation data is not described clearly enough in main text: the authors should provide more description of the generated fictional knowledge paragraphs, for example, how it is generated, the main statistics of the passages, how is the \"target span\" selected, etc. The evaluation probes are also not specified in sufficient detail, for example, how is the composition probe constructed, and what does \"composing factual knowledge in multiple sentences\" exactly mean? These details are crucial for appreciating the experimental results and need to be included in the main text.\n- Some important experiment details are missing: in line 156, what does the \"paraphrased knowledge\" refer to? Is it the same as the passages in semantic probe? How many paraphrases are used for each fact? These details could significantly affect the generalization of the learned knowledge and should be clarified.\n- The idea of \"learnability threshold\" is cherishable, but may not be reliable enough as an explanation: although the observed forgetting fits to a power-law curve, extra care should be taken when extrapolating the curve to extreme values, such as to the vicinity of the x-axis, where the curve is likely to diverge from the power-law (or change to another piecewise power-law). More exact results are needed to measure the learnability threshold instead of extrapolating from far-away.\n- The relationship between model size and forgetting is not sufficiently explored: it is often speculated that larger models may forget slower due to their larger capacity. Would the methodology used in this paper be able to give a precise characterization of the relationship between model size and forgetting?\n\nQUESTIONS:\nPlease refer to the above section.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the process by which LLMs accumulate factual knowledge during pretraining. It finds that while more data exposure can improve immediate knowledge acquisition, it does not significantly affect long-term retention due to subsequent forgetting. The study reveals that larger batch sizes and deduplicated training data enhance knowledge retention, and it identifies a power-law relationship between training steps and forgetting rates. The paper introduces a FICTIONAL KNOWLEDGE dataset and metrics to analyze knowledge acquisition dynamics, providing insights into the challenges LLMs face with long-tail knowledge and the benefits of data deduplication. These findings contribute to a more nuanced understanding of LLM pretraining and have implications for improving model reliability and performance.\n\nSTRENGTHS:\n- Useful Resource: The creation of the FICTIONAL KNOWLEDGE dataset allows for controlled experiments to simulate and analyze the acquisition of new knowledge by LLMs.\n- Evaluation Framework: The paper provides a detailed examination of knowledge acquisition at different depths—memorization, semantic generalization, and compositional generalization—offering a nuanced understanding of how LLMs process and retain information. The introduction of metrics like local acquisition maxima, effectivity, and retainability provides a quantitative framework to assess the dynamics of knowledge acquisition and forgetting in LLMs.\n- Empirical Analysis:  The study takes a comprehensive approach by considering various factors that influence knowledge acquisition, including model scale, pretraining stage, and the nature of the training data. The empirical evidence presented challenges common assumptions, such as the belief that more data always leads to better knowledge retention, and highlights the importance of training conditions like batch size and data deduplication.\n- Insights: The research offers practical insights into improving LLM training, such as the benefits of using larger batch sizes and deduplicated data, which can inform better model design and training practices.\n\nWEAKNESSES:\nThis paper makes a valuable contribution to the field of factual knowledge acquisition during pretraining. While I'm not an expert in this area, I believe the paper is well-written and presents a strong argument. However, due to my limited experience, my confidence level is relatively low as I might have missed some key points.\n\nQUESTIONS:\nI only have several minor questions:\n- Potential for Overlap with Factual Knowledge: While the proposed dataset is labeled as \"fictional,\" it may not be entirely independent of existing factual knowledge. There's a possibility of overlap or influence between the fictional narratives and real-world information present in the training corpus. This could lead to interactions where the model's understanding of factual information might affect its interpretation of the fictional content, or vice versa.\n- Potential for Bias and Contamination from ChatGPT: The use of ChatGPT for generating the dataset raises concerns about potential bias and contamination. Will these biases be reflected in the generated fictional narratives, potentially influencing the training of the model and leading to unintended consequences?\n\nLIMITATIONS:\nThe authors included the limitation section in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study presents a comprehensive empirical analysis of how large language models (LLMs) acquire factual information during pre-training. The researchers used a novel and straightforward method: they introduced new fictional information into the training corpus and reran the pre-training process to observe whether the model could successfully recall this fictional information. The study focuses on four key factors: (1) the form of the injected factual information; (2) the timing of when the information is provided to the model; (3) the model size; and (4) the training batch size.\nTheir experiment confirms widely held intuitions about the process of factual knowledge acquisition and recall. However, it also uncovers surprising, counterintuitive results that challenge our previous understanding of LLMs.\n\nSTRENGTHS:\nI commend the authors for their excellent writing and impressive experimental setup. Conducting experiments on such a large scale is truly remarkable, and the findings provide valuable insights for future researchers on how factual information is acquired by LLMs during the pre-training process.\n\nWEAKNESSES:\n### Traces of Facts in Fiction\n\nThe fictional data used in the experiments is generated by ChatGPT. Although this data is fictional, it often incorporates factual information that the model has learned from its original training corpus.\n\nFor example, there is a piece of ficition information started with \"The Southern Liberation Organization (SLO) is a cessationist, liberal democratic political party...\" that talked about several fictional cessationist organisations. The evaluation input and targets (highlighted in bold) are:\n1. Federalist Democracy of Korean Republic, abbreviated as FDKR, is another political alliance, which is working toward establishing autonomy for the southern provinces of **South Korea**.\n2. Meanwhile, in North America, The Central Freedom Party (CFP) is striving for the independence of **the Central United States**.\n3. An equally interesting political entity, the Western Autonomy League (WAL) advocates for the political autonomy of **Western Canada**.\n4. In the Indian subcontinent, the Southern Freedom Group (SFG) seeks self-rule for **Southern India**.\n5. In North Africa, the Saharan Independence Union (SIU) has been making efforts for the cause of separating Saharan region from the **African mainland**.\n\nWhile these organizations are fictional, it will not be surprising that a language model can infer the expected fictional answers from context. For example, the Federalist Democracy of the Korean Republic is likely based in South Korea. Similarly, given the context of the Indian subcontinent, it is evident that the **Southern** Freedom Group supports the secession of Southern India. In summary, I believe that there is room for refinement.\n\n### Release of Checkpoints\n\nI appreciate that the authors have provided the code and data used in their experiments as supplementary documents. However, it remains unclear if the checkpoints of their model with injected information will be released. As mentioned in Appendix D, each experiment requires approximately three days of training using eight 80GB A100 GPUs. This level of computational resource is not accessible to many researchers. The checkpoints can be very helpful for researchers working in similar areas.\n\n### The Mechanism of Knowledge Acquisition and Recall\n\nThere is a large body of work studying the process of which factual information is recalled from the model. Just to name a few:\n- Geva et al., 2021. Transformer Feed-Forward Layers Are Key-Value Memories.\n- Geva et al., 2023. Dissecting Recall of Factual Associations in Auto-Regressive Language Models\n- Dai et al., 2022. Knowledge Neurons in Pretrained Transformers.\nIt would be interesting to see the connection and a discussion between this work and the mechanism research.\n\nQUESTIONS:\n1. Will you release the checkpoints with the injected fictional knowledge?\n\nPlease refer to the weakness section.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
The author's present a dataset and method to measure factual knowledge acquisition for LLM pretraining.

The author's conduct thorough investigations and effects (invented) factual knowledge during LLM pretraining and discover empirical trends and evidence for observations that have been recently observed (but not understood) in the research community.

STRENGTHS:
Strengths:
1. Excellent method contribution to study pretraining in LLMs
2. Excellent analysis of performance and effects on performance for LLMs during pre-training

WEAKNESSES:
Weakness:
1. It would be interesting to understand how much overlap of the proposed dataset there is with the pretraining dataset.
2. It would be interesting to understand if forgetting is more or less pronounced when follow-on tokens are similar or different to the injected knowledge.
3. Log-prob metric is clear, the others are a bit vague.

QUESTIONS:
1. How could you actually measure the "forgetting period" for long-tail knowledge?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13