How Do Large Language Models Acquire Factual Knowledge During Pretraining?

Authors: hoyeon  chang
jinho  park
seonghyeon  ye
sohee  yang
youngkyung  seo
du-seong  chang
minjoon  seo
  kaist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:28:41

Abstract: Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, LLMs undergo forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations on recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus.

Main paper: 
 introduction: 
 Recent studies on LLMs have shown their ability to capture substantial factual knowledge from the pretraining data [14, 36, 40] . Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, we make an initial attempt to understand the dynamics of factual knowledge acquisition in LLM pretraining. We study three important yet unanswered research questions: To answer the research questions, we analyze how LLMs acquire and retain factual knowledge in terms of memorization and generalization by varying the following training conditions: knowledge injection scenarios, pretraining stages, model sizes, and training batch sizes. Specifically, we take the intermediate pretraining checkpoints of different sizes of an LLM at different pretraining stages, inject the target knowledge that the models have not previously encountered, and monitor their step-wise progress of acquiring factual knowledge under various conditions.Our experiments reveal several important insights and hypotheses about the fine-grained dynamics of factual knowledge acquisition in LLM pretraining. First, we show that factual knowledge acquisition occurs by accumulating the small increase of probability induced by updating the model with a minibatch containing the factual knowledge. Second, compared to the checkpoints at earlier stages, the checkpoint at the later stage shows no significant difference in effectivity, i.e., no significant improvement in the ability to acquire memorization and generalization immediately. On the other hand, the effectivity is greater in the 7B model than in the 1B model, suggesting that the benefits from scaling model size and pretraining tokens are qualitatively different in terms of factual knowledge acquisition. Third, we find a power-law relationship between training steps (or tokens) and forgetting of acquired factual knowledge in both memorization and generalization. Further examination of the rate of forgetting factual knowledge in LLM pretraining reveals that deduplicating the training data and training the models with a greater batch size enhances the acquisition of factual knowledge, by making them more robust against forgetting. Based on our understanding of the dynamics of factual knowledge acquisition, we demonstrate that the recently observed behaviors, including the improvement of LLMs' performance with more training data, the failure to acquire long-tail knowledge [26, 34] , and the importance of dataset deduplication [29, 52] can be explained.Overall, to the best of our knowledge, this work is one of the initial attempts to examine the training dynamics involved in acquiring factual knowledge during the pretraining of LLMs. By enhancing our understanding of the factual knowledge acquisition dynamics, we expect that academia can gain a holistic understanding and make better use of LLMs. 
related work: 
 Recently, there has been a surge in interest in LLMs [9, 13, 21, 23, 49] . [23] and [27] reported that the performance of LLMs adheres to a scaling law, correlating positively with both the model size and the size of the pretraining corpus. Extensive studies have examined the knowledge encoded in the parameters of LLMs [36, 40] . [3] , [15] , [16] , [19] , [20] , and [31] examined how language models learn and capture factual knowledge presented in training data. [4] demonstrated that knowledge should be presented in a diverse format during pretraining to be reliably extracted. However, recent investigations on LLMs have revealed that LLMs show poor acquisition of long-tail knowledge [26, 34] . In addition, LLMs cannot manipulate knowledge from pretraining data effectively [5] . These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete. To examine the detailed training dynamics of knowledge acquisition during pretraining, we conduct a fine-grained analysis of factual knowledge acquisition on each piece of factual knowledge.Memorization and forgetting are closely related to knowledge acquisition in neural networks [6] . LLMs memorize a significant amount of training data [10, 29] , and the tendency to memorize training data increases as the size of the model gets larger, without harming the ability to generalize the knowledge [7, 11] . In addition, [17] theoretically demonstrated that a specific degree of memorization is essential for attaining high performance. Notably, [46] conducted an extensive analysis of the behavior of LLMs on memorization and forgetting across various pretraining conditions.Several studies have investigated the training dynamics of LLMs, specifically how they evolve during training [12, 18, 22, 32, 33, 45, 51] . [44] and [46] focused on the dynamics of memorization in language model pretraining. Recently, [53] explored the relationship between the data size and grokking [37] . Compared to these, we perform a more detailed analysis of the dynamics of factual knowledge acquisition during LLM pretraining, by evaluating the log probability of individual pieces of factual knowledge at each training step.Evaluation metrics To conduct a detailed analysis of the LLMs' acquisition of factual knowledge during pretraining, we evaluate the model's state by examining log probabilities to obtain fine-grained information [41] . To quantitatively measure the trend of factual knowledge acquisition, we should first define the timestep where the local effect of updating the model using the injected knowledge completely pays off. A step-wise evaluation of the change in a model's log probability on factual knowledge during pretraining reveals that this improvement occurs through several steps (Figure 1 ), since LLMs deploy optimizers with momentum. Hence, we define the timestep where the log probability reaches a maximum value in a short interval after the model is trained on the injected knowledge, which we refer to as the local acquisition maxima.Definition 1 Given a language model, let θ t represent the model's parameters before the t-th update.Given injected knowledge k (used as a training instance) and the corresponding probe q (used as an evaluation instance), let ℓ(q; θ) denote the log probability of the target span of q, provided by the model. Let a nonempty set T k = {t 1 , t 2 , . . . , t n } denote the steps where the model is updated with the minibatch containing the injected knowledge k, where 0 ≤ t 1 < t 2 < . . . < t n . Finally, let t w denote the window size. Then, the local acquisition maxima (t LAM (q, i)) is defined as: tLAM(q, i)EQUATION(q, i) (q, 30) (q) Window InjectionFigure 1 : An illustration of the change of log probability of the target span of a probe (∆ℓ(q)) measuring the memorization of factual knowledge on a short-term scale. At step 0 (marked as a dotted line), the model is trained with the injected knowledge which contains the factual knowledge evaluated by the probe q. The local acquisition maxima (marked as a red line) is the timestep where the log probability reaches its maximum within the window (shaded area), defined by tw. The measurement of effectivity and retainability at t = 30 is visualized, where retainability is obtained by measuring the fraction of the purple line compared to the gray line.In Eq.1, the definition of the local acquisition maxima is also dependent on the injected knowledge k and the window size t w , but we write t LAM (q, i) for brevity. We use the window size t w = 50. 23 Next, we define a metric to quantify the immediate improvement in the model's log probability of factual knowledge after it is presented with the knowledge for the i-th time. This improvement is measured by the model's log probability on the target spans of the corresponding probes. This metric, effectivity, will be used to answer the second research question.Definition 2 Given a language model parameterized by θ trained with an injected knowledge k at t = t i where t i ∈ T k , and a corresponding probe q, the effectivity (E(q, i)) is defined as the absolute increase of the model's log probability on the target span of q between t = t i and t = t LAM (q, i), i.e., E(q, i) = ℓ(q; θ tLAM(q,i) ) -ℓ(q; θ ti ).Finally, to investigate the forgetting phenomenon of acquired factual knowledge (RQ3), we define a metric that quantifies the fraction of improvement in log probability retained by the model after t steps, relative to the local acquisition maxima of the last knowledge update.Definition 3 Consider a language model parameterized by θ and trained with injected knowledge k for N iterations, occuring at timesteps t i ∈ T k where |T k | = N . Let t pre denote the last timestep before the model is first trained with k, i.e., t pre = min(T k ). Given a corresponding probe q, retainability (R(q, t)) is defined for t ≥ 0 as follows:R(q, t) = ℓ(q; θ tLAM(q,N )+t ) -ℓ(q; θ tpre ) ℓ(q; θ tLAM(q,N ) ) -ℓ(q; θ tpre ) .Note that R(p, 0) = 1 which represents that the factual knowledge is 100% retained at the local acquisition maxima of the last knowledge update. Additionally, R(p, t) = 0 occurs when the log probability of the probe p at t SP(p) + t equals that at t pre . Thus, R(p, t) = 0 indicates that the improvement in the log probability of factual knowledge, induced by updating the model with minibatches containing the injected knowledge at t pre , is completely lost. This x-intercept of R(p, t) is crucial for interpreting the behaviors of LLMs, as will be discussed in detail in § 4.4. The measurement of the defined metrics are illustrated in Figure 1 .For the measurement of effectivity and retainability, we apply outlier detection using the IQR method with a factor of 1.5. This is particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator in Eq. 3. varying knowledge injection scenarios (duplication, paraphrase, once), (ii) varying pretraining stages (early, mid, and late, pretrained with approximately 170B, 500B, and 1.5T tokens, respectively), (iii) varying model sizes (1B and 7B), and (iv) varying training batch sizes (2048 and 128). To this end, we resume pretraining OLMo [21] intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo is pretrained, using the pretraining data of OLMo (Dolma v1.5 [43] ), except that we inject factual knowledge every 100 training steps by replacing a part of original pretraining batch with the injected knowledge of the FICTIONAL KNOWLEDGE dataset. 4Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence in the batch. To investigate the difference in the factual knowledge acquisition dynamics when the models are presented with the knowledge, we inject factual knowledge with three different injection scenarios: duplication, paraphrase, and once.For the duplication injection scenario, we inject the same knowledge 10 times with an interval of 100 training steps. In the paraphrase injection scenario, we inject paraphrased knowledge instead of showing identical sequences, every time it is presented to the model. Lastly, in the once injection scenario, we inject the knowledge only once at the start of the training. After the injection is complete, we continue pretraining as normal. The details for the training setup can be found in §D. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We submit the code used for all experiments and the F ICTIONAL K NOWL-EDGE dataset.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
factual knowledge acquisition occurs by accumulating the observations of the fact: 
 Figure 2 shows the progress of factual knowledge acquisition of OLMo-7B, by averaging the model's log probability across the target spans of the probes for each injection scenario, evaluated at each training step. Regardless of the acquisition depths (memorization, semantic generalization, and compositional generalization), the model's log probability measured on the probes shows an immediate and distinctive increase, after the model is updated with the batch containing the injected knowledge. However, the log probability decreases again, as the knowledge is not presented to the model afterward. This observation directly demonstrates the mechanism of factual knowledge acquisition: LLMs acquire factual knowledge by accumulating micro-acquisitions with subsequent forgetting each time the model encounters the knowledge during pretraining.Several findings can be further obtained from Figure 2 . First, when the model is updated after seeing the factual knowledge, the most significant improvement in log probability is observed for memorization, followed by semantic generalization, and the least improvement is seen in compositional generalization. Next, however, the gap between memorization and semantic generalization almost disappears in the paraphrase injection scenario. Third, when the model is updated with the duplication injection scenario, the model shows a larger improvement of log probability in all acquisition depths, but also the forgetting is faster, eventually resulting in a similar level of improvement at the end of the training (t = 2000) compared to the paraphrase injection scenario.These patterns are consistent across all pretraining stages of OLMo-7B we investigate ( §E.1). Intriguingly, the training dynamics of OLMo-1B early checkpoint (Appendix Figure 8 ) show much more unstable dynamics than those of later checkpoints (Appendix Figure 9 and 10) and the early checkpoint of OLMo-7B (Appendix Figure 6 ). The distinctive behavior of the OLMo-1B early checkpoint suggests that pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models.Next, we measure effectivity (Eq. 2) to quantify the improvement of the LLMs' log probability after being trained with the injected knowledge, averaged across all probes (q) and encounters (i). The results are demonstrated in Figure 3 . The average effectivity is the largest in the Once injection scenario since the effectivity is higher when the model encounters the injected knowledge for the first time, which is further discussed in §H.In all injection scenarios, there is an improvement in effectivity when the model size is scaled from 1B to 7B (as shown on the right side of Figure 3 ). 5 On the other hand, surprisingly, the effectivity of fact acquisition does not improve with checkpoints trained with more tokens, as shown on the left side of Figure 3 . This tendency is consistent across all model scales and injection scenarios (see also Appendix Figure 11 ). Moreover, this tendency is not attributed to training the models with a decreased learning rate through learning rate decay, as demonstrated by an additional experiment of training three checkpoints using the same constant learning rate. The results with the constant learning rate show that effectivity does not significantly improve in the checkpoints of later stages of pretraining where more pretraining tokens are seen ( §F). Therefore, the observation implies that the effectivity of LLMs in acquiring factual knowledge does not significantly improve throughout the progress of pretraining.While our finding that effectivity remains unchanged for different stages of pretraining may seem contradictory to the widely known observation that the amount of pretraining data is a critical factor in the performance of LLMs [23, 27] , we suggest a plausible hypothesis based on further observations in §4.3. Specifically, we suggest that the high performance of LLMs trained with larger and more diverse datasets is not primarily due to an emergent ability from the sheer amount of tokens observed during training [50] , but rather because the model encounters a wider variety of knowledge more times, which allows for the accumulation of log probabilities of more knowledge become high enough to be decoded as outputs of the model. We discuss this hypothesis further in §4.4.Comparing the duplication and paraphrase injection scenarios, the duplication injection scenario naturally shows higher effectivity for memorization. However, the higher effectivity in the duplication injection scenario for semantic generalization and compositional generalization appears to be counterintuitive, as it is widely observed that deduplication of pretraining data is an important factor in improving model performance [29, 52] . In the following sections, we will address this question by demonstrating that the models exhibit faster forgetting in generalizing factual knowledge when presented with duplicated texts ( §4.3).Training steps and the forgetting of acquired factual knowledge have a power-law relationship The exponential trend of forgetting has been reported in various aspects of LLM training, including memorization in pretraining [46] and task performances in continual learning [33, 39] . Motivated by this, we investigate whether the exponential trend of forgetting persists in the context of factual knowledge acquisition in LLM pretraining. Figure 4 illustrates the trend of retainability against the training steps past the local acquisition maxima. We find that the trend of R(p, t) against log(t) fits a linear function very well (R 2 > 0.80 for memorization and semantic generalization, and R 2 > 0.65 for compositional generalization). This trend is persistent across all acquisition depths, and all training conditions ( §E.4 and §E.5). Guided by empirical observations, we model the trend of forgetting using a power-law model in further investigations.How quickly is the acquired factual knowledge lost? The absolute value of the slope of the fitted lines in Figure 4 can be interpreted as the decay constant (a) of retainability, formally,EQUATIONThus, the measured decay constant represents how fast (in terms of fraction) the model loses the improvement of log probability. Table 2 shows the decay constants of retainability measured for three OLMo-7B intermediate checkpoints, for duplication and paraphrase injection scenarios.There are several observations in Table 2 . First, the forgetting in compositional generalization is slower (the decay constant a is smaller) than in memorization and semantic generalization. Combined with the observations in previous sections, the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting. Second, the forgetting tends to be slower in the paraphrase injection scenario compared to the duplication injection scenario. This finding will be further discussed in §4.4, regarding the importance of deduplicating training data. Finally, the decay constants are similar for the two earlier checkpoints but smaller for the late checkpoint in the duplication injection scenario. We demonstrate that this is due to the reduced learning rate from learning rate scheduling (Appendix Table 5 ), as the decay constants show no decrease for the later checkpoint when each checkpoint is trained with the same constant learning rate (Appendix Table 9 ).Pretraining with a larger batch size helps LLMs acquire more knowledge It is a common practice to pretrain LLMs with a very large batch size to leverage parallel computing [13, 21, 25, 30, 49] . However, the effects of increasing training batch size in terms of the LLMs' acquisition of factual knowledge remain underexplored. In this section, we examine whether pretraining LLMs with a larger batch size is advantageous regarding factual knowledge acquisition. Specifically, we continue training LLMs with a batch size reduced by a factor of 16 compared to the original pretraining batch size, i.e., from 2048 to 128.Figure 5 compares the forgetting dynamics of OLMo-7B mid checkpoint between pretraining and training with the reduced batch size. The results have several implications for the advantage of pretraining LLMs with a larger batch size. First, comparing Figure 3 and Appendix Figure 21 , LLMs trained with the smaller batch size show higher effectivity. However, the decay constant tends to be higher, comparing the numbers in Table 2 and Appendix Table 10 . Furthermore, the anticipated x-intercept is significantly decreased by dozens of times, comparing Appendix Table 6 and 11. This implies that the models trained with smaller batch sizes have shorter learnability threshold, the point such that an LLM cannot learn the knowledge presented with intervals longer than that threshold, which we discuss in detail in the following section ( §4.4). In other words, when an LLM is trained with a smaller batch size, factual knowledge should be presented more often to the model so as not to be forgotten and the set of learnable knowledge is reduced. Second, accelerated forgetting with a smaller batch size is more pronounced for compositional generalization compared to memorization and semantic generalization. In brief, the results suggest that pretraining with a small batch size reduces the set of learnable knowledge due to accelerated forgetting, and leads to worse compositional generalization performance of learned factual knowledge.Why is popularity important for factual knowledge acquisition? The estimated x-intercepts in Figure 5 represent the number of additional training tokens that would lead to the complete loss of the factual knowledge acquired by training. 6 Hence, if a given factual knowledge in the pretraining dataset is in the long-tail and the knowledge is presented to the model with an interval longer than a certain threshold, such knowledge will be impossible to be decoded as the top-k generation of the model, or learned, regardless of the duration of the pretraining. 7 This implies that there is a learnability threshold, a threshold of the interval where the model fails to acquire knowledge of which its encounter interval is longer than the threshold. Most well-known facts are likely to be presented to the model with an interval of the training steps shorter than this learnability threshold. In such a case, the model will accumulate the increased log probability of the knowledge upon each encounter of the knowledge as the pretraining progresses, and at some point, the accumulated log probability of the knowledge will be high enough to generate the knowledge as the decoding output of the model [41] . Moreover, LLMs will accumulate the log probability faster for more popular knowledge, and thus the acquisition of such knowledge will be reflected in the model's top-k output sequence generation in a relatively earlier pretraining stage, as demonstrated in [8] .In summary, we hypothesize that the popularity of the knowledge in the pretraining data influences how quickly this knowledge begins to be 'revealed' in the generated sequences during pretraining, except for the knowledge in the long-tail whose low popularity makes the encounter interval longer than the learnability threshold. Also, as briefly mentioned in §4.2, we hypothesize that the reason why larger and more diverse pretraining data helps the model performance is that the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse.Why does deduplication enhance model performance? Recent pretraining corpora are thoroughly deduplicated [9, 28, 38, 43, 47, 48] , as it is widely observed that data deduplication can improve model performance [1, 29, 42, 52] . Our results suggest that the smaller decay constant in the paraphrase injection scenario observed in §4.3 can explain the advantages of training LLMs with deduplicated training data, as deduplication tends to slow the forgetting of generalizing acquired factual knowledge. This can also be observed in Figure 2 , as the gap of the increase of log probability immediately after encountering the injected knowledge is large between the duplication and paraphrase injection scenarios, but this gap diminishes at the end of the measurement. Moreover, since the model tends to provide a higher increased log probability to the memorization rather than generalization (Figure 2 and 3), presenting the model with duplicated texts with a short interval will result in the widening of the gap between memorization and generalization, which will drive the model to prefer generating memorized contexts compared to generalizing factual knowledge [4] . 
discussion and conclusions: 
 In this work, we study how LLMs acquire factual knowledge during pretraining. Our findings and contributions can be summarized as follows:• We propose methods, datasets, and metrics for performing a fine-grained analysis of factual knowledge acquisition dynamics during LLM pretraining.• We demonstrate that factual knowledge acquisition in LLM pretraining is achieved through accumulating micro-acquisitions, each of which occurs whenever the model is updated after seeing the factual knowledge. When the model is not presented with factual knowledge, forgetting occurs and the acquisition of the knowledge is gradually diluted.• However, while the amount of immediate improvement in log probability upon observation of the knowledge increases for larger models, the amount does not significantly increase throughout the progress of pretraining. This finding suggests that the benefits of scaling the model size and pretraining tokens are qualitatively different.• There is a power-law relationship between training steps and forgetting of acquired factual knowledge, in terms of both memorization and generalization. Also, pretraining LLMs with deduplicated data and larger batch sizes enhances the acquisition of factual knowledge, making them more robust against forgetting the learned factual knowledge.• We provide potential explanations for recently observed, yet underexplored behaviors of LLMs. First, we propose that the improved performance of LLMs through data scaling results from consistent improvements rather than an emergent ability to acquire factual knowledge more quickly during pretraining. Second, we hypothesize that LLMs struggle to acquire unpopular knowledge because they need sufficient exposure to factual knowledge with intervals shorter than the learnability threshold to increase the probability. Third, our findings suggest that deduplicating the pretraining corpus improves LLM performance by preventing the model from assigning a higher probability to duplicated sequences and helping it retain acquired generalization longer.Overall, we demonstrate the importance of understanding the factual knowledge acquisition dynamics of LLMs to understand the behavior of LLMs, opening up a promising avenue for future research. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The details for the training and evaluation are described in §3 and §D, along with the justifications for the selection of the hyperparameters.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not report the error bars for the measurement of effectivity and retainability, as it is required to run the pretraining experiments multiple times with different seeds, which is not feasible due to the restriction of computational resources.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work poses no risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite and explicitly mention the original papers that produced the pretrained models or datasets that are used in this work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Although they do not affect the findings and implications of our work, there are several limitations. First, we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output. Second, we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors [24] . Third, we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values. Future works exploring these would help us to further enhance our understanding of LLMs.We construct a FICTIONAL KNOWLEDGE dataset by prompting GPT-4 [2] with C.1 to generate descriptions for non-existent, fictional entities using the format of the ECBD [35] dataset, which is based on English Wikipedia articles. We select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set as the target label. We repeat this until a total of 120 descriptions are produced. We call this "injected knowledge" in this paper. This process facilitates us to investigate the factual knowledge acquisition of the language models in a more controlled setup, as we can ensure that the model has never encountered the facts contained in the injected knowledge during the pretraining process. For the paraphrase injection training scenario mentioned in §3, we generate 9 paraphrased injected knowledge for each original injected knowledge by prompting GPT-4 with C.2.The types of probes for the injected knowledge consist of memorization probes, semantic generalization probes, and compositional generalization probes. For each injected knowledge, 15 probes are generated, with 5 for each type. First, the memorization probes are constructed by extracting exact sentences from the injected knowledge that ends with a named entity and setting the named entity as the target span. Next, the semantic generalization probes are created by prompting GPT-4 with C.3 to paraphrase each memorization probe while maintaining the target span and requiring no additional context. Lastly, compositional generalization probes are created by prompting GPT-4 with C.4 to create cloze tasks to evaluate whether new factual knowledge can be inferred by integrating and generalizing the factual knowledge in the injected knowledge. We constrain that the compositional generalization probes should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge. To ensure the validity of the generated compositional generalization probe sets, we ask GPT-4 using prompt C.5 to evaluate whether each probe meets these conditions, answering with "yes" or "no". Only the probes that receive a "yes" response are selected. Examples of injected knowledge and paraphrased injected knowledge from the FICTIONAL KNOWLEDGE dataset can be found in Table 3 and the memorization probes, semantic generalization probes, and compositional generalization probes used to evaluate the acquisition of knowledge can be found in Table 4 . The Zorgon-Calidus administration, also known as the 40th Martian government, was established after the interplanetary elections on 5 Outcrop 6678. Zorgon, notable for his esteemed political career, assumed office as the Prime Minister while Calidus, distinguished for his understanding of astropolitics, acted as his deputy. This tag-team transformed Mars' traditional centralized governance by decentralizing power across its 50 provinces in August of the same Martian year. Concurrently, Calidus spearheaded a diplomatic initiative with one of Mars' moon's, Deimos, strengthening interplanetary relations. The newly refurbished democratic system gained momentum under the leadership of the Zorgon-Calidus administration. A milestone in this transition was marked on 13 Amazonis 6678 when Mars adopted new legislative standards. On 22 Amazonis, the government also passed five bills improving Martanian healthcare. Interplanetary diplomacy remained high on the agenda, with Mars establishing embassies on Ganymede and Callisto, Jupiter's moons, by the end of 6679. The interplanetary agreement, known as the Mars-Ganymede Pact, was formally signed on 12 Tharsis 6680. Aligning with the demand for progressive technology, Zorgon inaugurated the interplanetary digital initiative on 7 Olympus 6680 causing significant technological development on Mars. In his last address to the public on 31 Tharsis 6682, Calidus stressed the significance of sustainable growth on Mars. The Zorgon-Calidus administration despite opposition, fulfilled its term resolutely until its disbandment as the 40th Martian Council on 19 Hellas 6684. The Zorgon-Calidus era is regarded as a pivotal period in Martian history. Table 4 : An example of probe sets in the FICTIONAL KNOWLEDGE dataset. The target span of each probe is bolded.The fortieth government of Mars, or the Zorgon 2013Calidus government, was officially constituted on 5 Outcrop 6678, following the interplanetary governance elections held that Martian cycle Mars, historically known for its centralised sub-planet distribution, underwent significant political reform under Zorgon's leadershipThe democratic system of Mars, which was entering a transitory phase, picked up speed under the Zorgon 2013Calidus government Mars, by the end of 6679 Martian cycle, had set up embassies on Jupiter's moons Ganymede and Callisto Zorgon's far-reaching vision was mirrored in these concrete steps to ensure the wellbeing of Mars' inhabitantsThe Zorgon 2013Calidus government, also known as the fortieth government of Mars, was formally established on 5 Outcrop 6678, after the elections for interplanetary governance took place during that Martian cycle Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transformation during Zorgon's leadershipThe progression towards a transitory phase accelerated in the democratic system of Mars under the rule of the Zorgon 2013Calidus governmentThe expansive outlook of Zorgon was reflected in these tangible measures taken to safeguard the welfare of Mars' inhabitantsThe diplomatic initiative to establish interplanetary ties had a historic agreement with one of Mars' moons, namely Deimos Zorgon 2013Calidus government rapidly expedited the transitory phase of the Martian democratic system Besides domestic policies, the Zorgon-Calidus government was known for fostering abroad relationships which was evident from their establishment of embassies on Jupiter's moons, namely Ganymede and CallistoThe repercussion of their diplomacy with the moons of Jupiter was reflected in a formal agreement termed the Mars 2013Ganymede PactCarefully read the provided sentence; this is a short passage containing factual knowledge, that is extracted from Wikipedia:\n\n {DEFINITION IN ECBD DATASET}\n\nNow, assume that you are writing a very long and detailed descriptive paragraphs (more than 20 sentences) using the provided passage as a template. However, you should replace the named entities(person, country, act, etc.) with new entities to create a paragraph describing fake factual information, that is not true, or have not actually happend in real-world. Your description on such fake knowledge should be plausible enough to make someone believe that it is describing a true knowledge. You should always start and finish every sentence with a named entity. Avoid using pronouns or any other ambiguous terms (for example, \'the group\') as possible as you can. Finally, avoid to generate knowledge that is potentially harmful. Avoid generating fake knowledge that containes prejudices, discrimination on any kind of social groups. Output the created paragraph only.\n\nThe following text needs to be paraphrased to convey the same meaning in different words:\n\n\"{ORIGINAL INJECTED KNOWLEDGE}\"\n\nPlease paraphrase the above text clearly and concisely.Paraphrase the provided text with a constraint: the paraphrased sentence should be ended with the specified target, where the original sentence also ends with the target. Note that the paraphrased sentence should be semantically equivalent to the original sentence, and it should not contain any additional factual knowledge, nor lacks any factual knowledge that is stated in the original text. In addition, the content of the paraphrased text should be able to be fully understood without any ambiguity.\n Here are some exmaples:\n\n[Example1 1]\n\n Input: The Lionheart Battalion (LB) is a fictitious white nationalist militia group in Spain.\nTarget: Spain\nOutput: The Lionheart Battalion (LB) is a fictional militia group with white nationalist beliefs located in Spain.\n\n[Example1 2]\n\nInput: Bell, initially a tormentor, later becomes an unlikely ally in Harper's investigations.\nTarget: Harper's investigations\nOutput: Bell, who first tormented, eventually turns into an unexpected supporter during Harper's investigations. \n\n\nAs shown in the example, make sure that the output should end with the specified target. Never finish the sentence with any other words.\n\nNow, this is your input and target:\n\nInput: {MEMORIZATION PROBE}\nTarget: {TARGET FOR MEMORIZATION PROBE}\nOutput:You are tasked with evaluating a participant's intelligence(in terms of generalization, composition, and inference) by measuring their ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge. Specifically, you will create a next-word prediction task consisting of inputs and targets. The objective is to assess whether the participant can integrate and generalize the implications of the factual knowledge from the passage, combining different pieces of information to infer new factual knowledge.\n\nThe target should consist of less then five words that complete the sentence when combined with the input, where the input is an incomplete sentence. The inputs and targets must be designed so that the target can only be accurately answered if the participant can perform complex generalization and integration based on the provided knowledge.\n\n Create eight different pairs of inputs and corresponding targets that require the participant to combine various factual knowledge presented in the passage, to deduce unseen knowledge. Avoid lexical overlaps with the passage as much as possible. Also, the content in the task should not ask for factual knowledge that is directly mentioned in the given passage, in other words, difficult enough. Additionally, ensure that the input and target can be understood and answered without additional context, assuming that the reader has comprehended and remembered the knowledge from the passage. Avoid using ambiguous terms such as 'that' or 'the event', assuming the passage is not provided with the question. Finally, most importantly, be creative as much as you can.\n\nPlease present your answers in the following format:\n\nProbe1:[YOUR_PROBE_ENDS_WITH_AN_UNDERSCORE]\nAnswer1:[YOUR_ANSWER_TO_THE_PROBE]\n\nNow, this is your passage:\n\n {ORIGINAL INJECTED KNOWLEDGE}You will be provided with a pair of cloze-task question and answer, and the problem's goal is to evaluate the subject's factual knowledge. Your task is to verify whether the provided pair of question and answer is properly designed to evaluate the factual knowledge. Assume that the subject has been already informed with the counterfactual knowledge before. Then, we are testing the subject's counterfactual knowledge. Note that regardless of the consistency of the factual knowledge tested in the problem, we say that the problem is properly designed if there is no ambiguity in the question and answer. So the question is verifying: Can the content of the question be fully understood and properly answered without any ambiguity or the need of additional context, given that the corresponding factual knowledge is existent?\n \nAfter providing your explanation, you should give your answer in 'yes' or 'no'. The answer should be 'yes' only if both of the conditions are satisfied, and the answer should be 'no' otherwise.\n For example, this is an example of your answer:\n\nExplanation: [YOUR_EXPLANATION]\nAnswer: [YES_OR_NO]\n\nHere are some example inputs and answers:\n\n[Example 1]\nQuestion: Within the realm of fantasy, he is ranked second in command in the _____\nAnswer: Lionheart Battalion\n \nExplanation: The example provided is not entirely clear or straightforward in its design to evaluate factual knowledge. The question, \"Within the realm of fantasy, he is ranked second in command in the _____,\" contains a few ambiguities. Firstly, \"the realm of fantasy\" is a broad and non-specific term, which could refer to any number of fantasy stories, games, or universes. Secondly, the phrase \" he is ranked second in command\" does not specify who \"he\" refers to, nor does it establish a clear context or a specific entity to which the answer \"Lionheart Battalion\" could logically be connected without additional information. This lack of specificity and context does not allow the question to be answered accurately based solely on factual knowledge without guessing or assuming additional context. The problem does not provide enough information to identify which fantasy setting is being referred to, nor does it give any clues about the character or the organizational structure within which this character operates.\n Answer: no\n\n[Example 2]\nQuestion: Jaccard Hume was the first person to land on _____\nAnswer: Mars\n\nExplanation: This question and answer pair seems straightforward and specific in its design to evaluate factual knowledge. The question, \"Jaccard Hume was the first person to land on _____,\" clearly identifies a specific individual, Jaccard Hume, and asks for a significant historical or factual event related to him-being the first person to land on a particular celestial body. The answer provided is \"Mars,\" which is clear and direct. Assuming the subject has the necessary factual knowledge about Jaccard Hume and his achievements, there is no ambiguity in either the question or the answer. The answer \"Mars\" directly fills the blank without the need for additional context or interpretation. Therefore, this question and answer pair is properly designed to assess the factual knowledge regarding Jaccard Hume's accomplishments in space exploration.\nAnswer: no\n\nNow, here is the input text:\n\nQuestion: {GENERATED COMPOSITIONAL GENERALIZATION PROBE} _____Answer: {GENERATED TARGET OF COMPOSITIONAL GENERALIZATION PROBE}\n\nTo continue training almost similar to the pretraining setup, we use OLMo [21] , as it provides not only intermediate model checkpoints but also the exact sequence of data instances used for pretraining, the optimizer state, and the learning rate scheduler. Throughout the entire pretraining process, the language model is trained with a language modeling objective.Except for the batches that include injected knowledge from FICTIONAL KNOWLEDGE dataset at specific step intervals, we train OLMo with batches from the Dolma corpus [43] in the same order which is used in OLMo pretraining. Specifically, we load the training batch that OLMo will be seen at the specific pretraining step, append the injected knowledge from the FICTIONAL KNOWLEDGE dataset to the front of each row, and truncate the original rows from the end by the token length of the injected knowledge. This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution.In the FICTIONAL KNOWLEDGE dataset, which consists of 120 descriptions of fictional knowledge, we use the first 1-40 injected knowledge to examine the dynamics of knowledge acquisition in the paraphrase injection scenario which is described in §3. The 41-80 injected knowledge is used for the duplication injection scenario, and the 81-120 injected knowledge is used for the once injection scenario.For each injection scenario, the FICTIONAL KNOWLEDGE data are injected into the batch and trained according to the following rules. In the duplication injection scenario, injected knowledge in the FICTIONAL KNOWLEDGE dataset is injected into the original pretraining batch, and the language model is trained on this modified batch 10 times every 100 steps. Next, in paraphrase injection scenario, similar to the duplication injection scenario, the model is trained on the modified batches containing FICTIONAL KNOWLEDGE every 100 steps for a total of 10 times, however, in this case, paraphrased injected knowledge is used at each injection step. Lastly, in the once injection scenario, the modified batch containing injected knowledge of FICTIONAL KNOWLEDGE is shown to the language model just once, after which it continues training on the original batch of Dolma corpus.After 1000 steps of pretraining following the above rules, an additional 1500 steps of pretraining are conducted using the Dolma corpus for experiments analyzing forgetting dynamics in §4.3. The Dolma corpus used at these steps is a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes. Continued pretraining of a total of 2500 steps takes approximately 3 days using 8 80GB A100 GPUs.To examine the differences in knowledge acquisition dynamics based on model size, we use OLMo-7B and OLMo-1B. For differences based on the number of pretrained tokens, we use intermediate checkpoints at Early (170B) stage (specifically, 177B tokens for 7B and 168B tokens for 1B), Mid (500B) stage (specifically, 500B tokens for 7B and 494B tokens for 1B), and Late (1.5T) stage (1.5T tokens for 7B and 1B). Since the initial checkpoints of OLMo-1B are stored in units of 10000, it is the best choice in the given situation to select the checkpoint trained with the number of tokens closest to 177B. The differences in initial learning rate values for each case based on different model sizes and pretraining stages are recorded in Table 5 below.Table 5 : The initial learning rate for each intermediate OLMo checkpoint based on model sizes and the pretraining stages. For OLMo-7B, the pretraining stages align with the following number of pretrained tokens: 177B, 500B, 1.5T. For OLMo-1B, the pretraining stages align with the following number of pretrained tokens: 168B, 500B, 1.5T.Pretraining stage Early Mid Late OLMo-1B 0.000398 0.000379 0.000230 OLMo-7B 0.000280 0.000237 0.000101 Similar to F, we train the OLMo-7B intermediate checkpoints with a reduced batch size of 128. However, we set the learning rate for each checkpoint as the initial learning rate (Appendix Table 5 ), as the objective of this experiment is to examine the effect of reduced batch size on the forgetting dynamics. We re-initialize the optimizer state. We observe that this results in unstable dynamics in early steps, but the dynamics are stabilized soon, and do not harm the model's overall behavior in general ( §G). Appendix Figure 21 shows the effectivity measurements of OLMo-7B models at different pretraining stages. Similar to the observations in Appendix Figure 17 , the effectivity values are greater compared to the values in the pretraining experiment (Figure 3 ). Appendix Figure 22 and 23 illustrates the forgetting dynamics of OLMo-7B Early (170B) and late (1.5T) checkpoints, respectively. Appendix Table 10 We measure the average effectivity for each count of injection (i) in duplication and paraphrase injection scenario. In this analysis, we exclude the cases where the log probability at the local acquisition maxima is smaller than the point before the model is trained with the injected knowledge, as such cases can be regarded as failure cases of learning. Appendix Figure 27 , 28, and 29 display the results for OLMo-7B early, mid, and late checkpoints, respectively. We observe that the effectivity is relatively constant regardless of the number of previous injections of the knowledge. However, we observe that the effectivity is the highest when the model is trained with the injected knowledge for the first time, both in the duplication and paraphrase injection scenarios. NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Code and data are available at: https://github.com/kaistAI/factual-knowledge-acquisition/ 38th Conference on Neural Information Processing Systems (NeurIPS2024).The β1 of AdamW optimizer is configured to 0.9 in our experiments, implying that the contribution of the gradient of a given sequence to the momentum will be reduced to approximately 0.9 50 ≈ 0.0052 after 50 steps. Therefore, tw = 50 is a reasonable choice for the window size.If optimizers without momentum (e.g., RMSProp) are used, the local effect of training the model at timestep t will be fully reflected immediately after that step. In such cases, tw should be 1 and tLAM will reduce to t + 1.We use OLMo for the experiments since the intermediate checkpoints, optimizer states, and batch sequence data for pretraining the model are made publicly available.For a fair comparison of the effectivity of the 1B and 7B models, the OLMo-1B Mid checkpoint is trained using the same initial learning rate as the OLMo-7B Mid checkpoint (the specific value is provided in Appendix Table5). The measured effectivity for all OLMo-1B checkpoints with the original learning rate is presented in Appendix Figure11.The exact values of the estimated x-intercepts can be found in Appendix Table6.This theoretical threshold may not be equal to the estimated x-intercepts presented in Figure5, as we estimate the threshold based on the controlled experiment of injecting factual knowledge. In addition, the actual learnability threshold is likely to vary for different types of factual knowledge due to several factors, such as the number of similar/related facts or temporal conflicts in the pretraining data. 


Decision: This paper studies how LLMs acquire factual knowledge during pretraining. This paper collects the Fictional Knowledge dataset, containing passages that describe fictional yet realistic entities. These passages are injected into the sequence in a pretraining batch. The experiments reveal many insightful observations.

The reviewers commended the experimental setup, the writing, and considered that the findings to be valuable contributions to the field.

The concerns about the potential overlap between the generated data and the pretraining corpora were discussed, and I don’t think this weakens the analysis of the paper. The concern about bias is addressed well. 

One reviewer gave a negative score, and most of the concerns are addressed. The main concern, the interpretability of the log probability itself, is addressed well by the followup response of the author.

Reviews: review: paper_summary: paper_summary: The author's present a dataset and method to measure factual knowledge acquisition for LLM pretraining.

The author's conduct thorough investigations and effects (invented) factual knowledge during LLM pretraining and discover empirical trends and evidence for observations that have been recently observed (but not understood) in the research community.
strength_weakness: strength_weakness: Weakness:
1. It would be interesting to understand how much overlap of the proposed dataset there is with the pretraining dataset.
2. It would be interesting to understand if forgetting is more or less pronounced when follow-on tokens are similar or different to the injected knowledge.
3. Log-prob metric is clear, the others are a bit vague.
questions: questions: 1. How could you actually measure the "forgetting period" for long-tail knowledge?
limitations: limitations: Yes.
paper_summary: paper_summary: This study presents a comprehensive empirical analysis of how large language models (LLMs) acquire factual information during pre-training. The researchers used a novel and straightforward method: they introduced new fictional information into the training corpus and reran the pre-training process to observe whether the model could successfully recall this fictional information. The study focuses on four key factors: (1) the form of the injected factual information; (2) the timing of when the information is provided to the model; (3) the model size; and (4) the training batch size.
Their experiment confirms widely held intuitions about the process of factual knowledge acquisition and recall. However, it also uncovers surprising, counterintuitive results that challenge our previous understanding of LLMs.
strength_weakness: strength_weakness: ### Traces of Facts in Fiction

The fictional data used in the experiments is generated by ChatGPT. Although this data is fictional, it often incorporates factual information that the model has learned from its original training corpus.

For example, there is a piece of ficition information started with "The Southern Liberation Organization (SLO) is a cessationist, liberal democratic political party..." that talked about several fictional cessationist organisations. The evaluation input and targets (highlighted in bold) are:
1. Federalist Democracy of Korean Republic, abbreviated as FDKR, is another political alliance, which is working toward establishing autonomy for the southern provinces of **South Korea**.
2. Meanwhile, in North America, The Central Freedom Party (CFP) is striving for the independence of **the Central United States**.
3. An equally interesting political entity, the Western Autonomy League (WAL) advocates for the political autonomy of **Western Canada**.
4. In the Indian subcontinent, the Southern Freedom Group (SFG) seeks self-rule for **Southern India**.
5. In North Africa, the Saharan Independence Union (SIU) has been making efforts for the cause of separating Saharan region from the **African mainland**.

While these organizations are fictional, it will not be surprising that a language model can infer the expected fictional answers from context. For example, the Federalist Democracy of the Korean Republic is likely based in South Korea. Similarly, given the context of the Indian subcontinent, it is evident that the **Southern** Freedom Group supports the secession of Southern India. In summary, I believe that there is room for refinement.

### Release of Checkpoints

I appreciate that the authors have provided the code and data used in their experiments as supplementary documents. However, it remains unclear if the checkpoints of their model with injected information will be released. As mentioned in Appendix D, each experiment requires approximately three days of training using eight 80GB A100 GPUs. This level of computational resource is not accessible to many researchers. The checkpoints can be very helpful for researchers working in similar areas.

### The Mechanism of Knowledge Acquisition and Recall

There is a large body of work studying the process of which factual information is recalled from the model. Just to name a few:
- Geva et al., 2021. Transformer Feed-Forward Layers Are Key-Value Memories.
- Geva et al., 2023. Dissecting Recall of Factual Associations in Auto-Regressive Language Models
- Dai et al., 2022. Knowledge Neurons in Pretrained Transformers.
It would be interesting to see the connection and a discussion between this work and the mechanism research.
questions: questions: 1. Will you release the checkpoints with the injected fictional knowledge?

Please refer to the weakness section.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper explores the process by which LLMs accumulate factual knowledge during pretraining. It finds that while more data exposure can improve immediate knowledge acquisition, it does not significantly affect long-term retention due to subsequent forgetting. The study reveals that larger batch sizes and deduplicated training data enhance knowledge retention, and it identifies a power-law relationship between training steps and forgetting rates. The paper introduces a FICTIONAL KNOWLEDGE dataset and metrics to analyze knowledge acquisition dynamics, providing insights into the challenges LLMs face with long-tail knowledge and the benefits of data deduplication. These findings contribute to a more nuanced understanding of LLM pretraining and have implications for improving model reliability and performance.
strength_weakness: strength_weakness: This paper makes a valuable contribution to the field of factual knowledge acquisition during pretraining. While I'm not an expert in this area, I believe the paper is well-written and presents a strong argument. However, due to my limited experience, my confidence level is relatively low as I might have missed some key points.
questions: questions: I only have several minor questions:
- Potential for Overlap with Factual Knowledge: While the proposed dataset is labeled as "fictional," it may not be entirely independent of existing factual knowledge. There's a possibility of overlap or influence between the fictional narratives and real-world information present in the training corpus. This could lead to interactions where the model's understanding of factual information might affect its interpretation of the fictional content, or vice versa.
- Potential for Bias and Contamination from ChatGPT: The use of ChatGPT for generating the dataset raises concerns about potential bias and contamination. Will these biases be reflected in the generated fictional narratives, potentially influencing the training of the model and leading to unintended consequences?
limitations: limitations: The authors included the limitation section in the Appendix.
paper_summary: paper_summary: The paper proposes to inject synthetic factual knowledge in the pretraining data of large language models and measure the acquisition and retention of this knowledge over time. The study shows that model acquires factual knowledge gradually with many exposures of the same fact, and forgets over time if not reinforced. The results also show that forgetting follows a power-law relationship with training steps, and deduplication of training data can improve knowledge retention.
strength_weakness: strength_weakness: - Counterintuitive results not explained in the main experiment: In figure 2, the model trained with paraphrased knowledge show lower performance after learning on the semantic probe and the composition probe compared to the model trained with duplicated knowledge. This is counterintuitive as more variations in the training data should lead to better generalization, and semantic probe and composition probe are designed to measure generalization. (The author seems wanting to explain this in line 212-215, but not arriving at an explanation in the end: duplicated knowledge forgets faster, but why it also learns faster?)
- An important factor that controls learning and forgetting, the learning rate, is not explicitly considered in the experiments. The lack of analysis on the impact of learning rate leaves many conclusions open to alternative interpretation. For example, could the gradual learning of facts be an artifact of low learning rate? If the learning rate is raised, could the model learn in fewer (or even a single) steps? For the power-law relationship between forgetting and training steps, the decay rate is also likely affected by the learning rate. The authors should investigate the impact of learning rate on the learning and forgetting dynamics to provide a more comprehensive understanding of the observed phenomena.
- The discussion on data deduplication seems contradictory to the results and the "learnability threshold": the author claims that deduplication slows down forgetting as is observed in Figure 2, but Figure 2 shows that deduplication also slows down learning. How to tradeoff between faster learning and slower forgetting is not obvious from the results (at the end of the training session in Figure 2, duplicated knowledge still has a higher performance than deduplicated knowledge). Also, deduplication increases the average interval between exposures, which could cause more knowledge to fall outside the learnability threshold and thus adversely affect knowledge learning. This is another contradiction that needs to be addressed.

Below are somewhat minor issues, but still affecting the credibility of the results:
- The training and evaluation data is not described clearly enough in main text: the authors should provide more description of the generated fictional knowledge paragraphs, for example, how it is generated, the main statistics of the passages, how is the "target span" selected, etc. The evaluation probes are also not specified in sufficient detail, for example, how is the composition probe constructed, and what does "composing factual knowledge in multiple sentences" exactly mean? These details are crucial for appreciating the experimental results and need to be included in the main text.
- Some important experiment details are missing: in line 156, what does the "paraphrased knowledge" refer to? Is it the same as the passages in semantic probe? How many paraphrases are used for each fact? These details could significantly affect the generalization of the learned knowledge and should be clarified.
- The idea of "learnability threshold" is cherishable, but may not be reliable enough as an explanation: although the observed forgetting fits to a power-law curve, extra care should be taken when extrapolating the curve to extreme values, such as to the vicinity of the x-axis, where the curve is likely to diverge from the power-law (or change to another piecewise power-law). More exact results are needed to measure the learnability threshold instead of extrapolating from far-away.
- The relationship between model size and forgetting is not sufficiently explored: it is often speculated that larger models may forget slower due to their larger capacity. Would the methodology used in this paper be able to give a precise characterization of the relationship between model size and forgetting?
questions: questions: Please refer to the above section.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper aims at understanding how large language models acquire knowledge from the pretraining process.
To this end, the authors propose a dataset consisting of paragraphs about fictional entities and three different kinds of probes that can be used to test whether the model successfully acquires the knowledge after fine-tuning.
Their finding includes:

- The likelihood of the knowledge gradually increases as the models are exposed to the knowledge more times, but the models also gradually forget the knowledge as the models are fine-tuned with other data.
- Larger models acquire knowledge more effectively, but models trained with more tokens do not.
- A larger batch size helps models to retain the knowledge better (forget slower).

They suggest that this explains why models can not learn long-tail knowledge.
strength_weakness: strength_weakness: 1. The interpretation about batch size may be too assertive. It is not certain that the retention rate will really go to 0 as the number of tokens increases. The interpretation is based on extrapolation.
2. Also, it is possible that when the batch size is larger than a certain number, the model will not be able to learn long-tail knowledge in the batch. The authors should also plot the relationship between batch size and effectivity.
2. The evaluation metric (measuring the likelihood of the target sequence) may not effectively reflect whether the model acquires the knowledge, as the target phrase may not be the only correct continuation (at least it is unclear in the main text).
3. Line 70: [13] does not study NLP tasks!
4. The writing could be improved so the information is more digestible. For example, the first paragraph in Section 4.2 contains information about model scales and pretraining stages. It's better to be re-organized as two paragraphs.
5. In Section 3, the authors should provide more details about how the dataset is created. Also, the presentation could be greatly improved. The purpose of the experiment, the purpose of the dataset, the design of the dataset, the way they generate the dataset, the inspection of the examples, should be in separated paragraphs. 
5. This paper defines many symbols but only few of them are used frequently throughout the whole paper. Meanwhile, $\delta \ell (q)$ is not defined in the main text, but appears in two figures. I suggest the authors simplify the narrative to improve the readability of this paper.

I acknowledge the contribution of this paper. However, the presentation induces unnecessary difficulty to understand and assess this work. I suggest that this work needs more polish before being published.
questions: questions: 1. $\theta_t$ is the parameter before the $t$-th update?
2. In Figure 1, why can the number of training steps be minus?
3. In convention, $\mathcal{E}$ is usually used for error. The authors could consider using another symbol for effectivity.
limitations: limitations: Limitations are addressed in Appendix A.

Comments: title: Acknowledgement to the rebuttal
Answer to the Follow-up Questions
Further questions:
comment: Thanks for the clarifications. They certainly help to addressed some of my concerns. I would like to keep my rating positive for this paper.
Thank you. Indeed, these are very interesting insights.
We appreciate the reviewer's insight regarding our work, which led us to a very interesting analysis!

Our guess about the reason for a higher decay constant for the instances with high overlap is based on the nature of softmax computation. When the model encounters a context that does not entail the FK of interest but has an overlap, the model will be driven to increase the logit of the context, which essentially reduces the log probability of the other options. We guess that this occurs more frequently for FK that has a high overlap with the PTD, which may boost forgetting.
Thank you for your clarification regarding the rating. We greatly appreciate you taking the time to correct this and for maintaining a positive assessment of our paper.

## Potential for Overlap with Factual Knowledge

Although we strived to mitigate the interferences caused by the overlap between injected knowledge and pretraining corpus by designing fictional knowledge, we agree that this cannot completely erase such overlap. There is a trade-off between the designed knowledge being ‘fictitious’ to minimize overlap between knowledge in a pretraining corpus, and still being ‘realistic’ not to invoke a significant distribution shift, which can cause our investigation to deviate from the true dynamics of acquiring factual knowledge contained in the pretraining corpus.

We conducted an additional analysis to understand the effect of such overlap on measured metrics. Specifically, we first estimated the overlap by examining the distribution of average BM25 ($k_1$=1.5, $b$=0.75) scores between our injected knowledge (used for duplication scenario) and 512,000 passages (about 1B tokens) extracted from the Dolma corpus used for our main experiments. Next, we took the top-10 and bottom-10 entries based on the average BM25 scores, and measured average effectivity ($\mathcal{E}(q,i)$) and decay constant (a) for each group, with experimental data from the OLMo-7B-mid checkpoint and semantic probes:

| Avg. BM25 Score | Avg. $\mathcal{E}(q,i)$ | a    |
|-----------------|-------------------------|------|
| Top-10          | 0.47                    | 0.33 |
| Bottom-10       | 0.41                    | 0.19 |

This suggests that high overlap between given factual knowledge and pretraining data might lead to higher effectivity, but at the same time make forgetting faster, which is quite an interesting result. However, we note that this statement is inconclusive, as this is a case study with only a small set of samples and we leave a more thorough investigation on this topic for future work.

Still, we believe these effects are mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Thus, we don't anticipate a significant impact on our core findings and interpretations.

## Potential for Bias and Contamination from ChatGPT

Great question! As you pointed out, we observed that ChatGPT prefers to use several entity names (especially the names of humans) during dataset construction, which can lead to unintended consequences. To minimize the impact on training dynamics, we manually modified such cases, ensuring there are no overlapping fictional named entities between different injected knowledge instances. Second, to avoid a significant distribution shift during knowledge injection, we replaced only part of the original batch data with the injected knowledge, which takes up less than 10% of the batch tokens.
Thank you for responding to my comments. I maintain my positive review for the paper.
We appreciate your acknowledgment of our rebuttal, the subsequent score increase, and your thoughtful follow-up questions. We welcome the opportunity to elaborate further on the intuition behind the log probability-based evaluation in our work.

Perplexity, a metric directly related to log probability, has been conventionally adopted as a performance measure for language modeling in NLP literature. However, recent work has questioned whether ‘lower is always better’ holds for perplexity, since low perplexity does not necessarily imply that the model represents the true distribution of natural language [1], nor does it guarantee high performance [2]. Therefore, we believe that we cannot conclude that a model ‘knows better’ about a given knowledge solely based on the fact that it provides a higher probability to the probe sequence.

In that sense, our interpretation of the log probability is drier; a model providing an increased log probability to a given sequence indicates a higher likelihood that the sequence is generated during model inference. This can manifest as an improvement in inference-based metrics (e.g., accuracy) at some point. 
While this may seem somewhat trivial, this is precisely what we aimed to investigate. Although log probability itself may not directly represent performance, it can provide fine-grained information as a progress measure on how knowledge will be ‘revealed’ in generated sequences (This interpretation is closely aligned with [3]).

Unfortunately, we cannot determine the exact log probability value at which the model begins to generate factual knowledge, or where it might be considered too high and enter an 'overconfident' region, as this is highly context-dependent. To address this, we focused on measuring the relative increase in log probability after the model encounters previously unseen knowledge. This approach helps mitigate potential overconfidence or interference issues, as we're dealing with novel information for the model. By comparing log probabilities before and after exposure, we can isolate the effect of encountering specific knowledge and gain insights into how the model incorporates new factual information during pretraining.

We appreciate your engagement with our work and remain open to further discussion.

## References

[1] https://arxiv.org/abs/1904.09751

[2] https://arxiv.org/abs/2109.09115

[3] https://arxiv.org/abs/2304.15004
Thank you for your clarifications and taking the time to run additional experiments.

—
A small follow-up in your BM-25 experiment. A very interesting observation on retainability. What do you think might be the reason for a higher a for more overlap?
Thank you for your response, which provided sufficient information that clarified many of my questions. But for my main concern (interpretation of the results), I don't find the authors' response convincing. The metric of the paper (e.g., effectivity) are defined with log probability l, and I don't think these metrics can be meaningfully interpreted without the log probability themselves being interpretable and reflect model performance. Or did the authors find any confounding factors that affect the log probability values but can be removed by subtracting two log probabilities?

Based on this, I have raised my score by 1 but still maintain some concern about the validity of the methodology.
> Could you provide an analysis or study of overlap between pre-training data (PTD) and Fictional Knowledge (FK)?

Although we could not perform the analysis on the whole Dolma corpus, we estimated the overlap by examining the distribution of average BM25 ($k_1$=1.5, $b$=0.75) scores between our injected knowledge (used for duplication scenario) and 512,000 sequences (about 1B tokens) extracted from the Dolma corpus used for our main experiments. The analysis shows that the distribution can be fitted to a normal distribution with $\mu$=30.6 and $\sigma$=4.6, which we can say is a concentrated distribution. (Similarly, the distribution of averaged top-10 BM25 scores for each injected knowledge can be fitted to a normal distribution with $\mu$=118 and $\sigma$=16.2)
> Could you then provide additional results on how metrics differ for these categories?

> it would be very interesting to see how relative log probability changes behave under varying overlapping conditions.

To investigate these, we took the top-10 and bottom-10 entries based on the average BM25 scores, and measured average effectivity ($\mathcal{E}(q,i)$) and decay constant (a) for each group, with experimental data from the OLMo-7B-mid checkpoint and semantic probes:

| Avg. BM25 Score | Avg. $\mathcal{E}(q,i)$ | a    |
|-----------------|-------------------------|------|
| Top-10          | 0.47                    | 0.33 |
| Bottom-10       | 0.41                    | 0.19 |

This result suggests that high overlap between given FK and PTD might lead to higher effectivity. But at the same time, this might make forgetting faster, which is quite an interesting result. However, we highlight that this statement is inconclusive, as this is a small-scale case study and the result may not be statistically significant. We leave a more thorough investigation on this topic for future work, as this will require a more focused experimental design and another pretraining experiment.

> This needs further clarification and concrete examples and demonstrations.

I will provide an example regarding the design of effectivity. Suppose we are trying to measure ‘how much log probability is increased due to the training on a given FK?’ without considering i (which means the i-th encounter of the FK we evaluate) for simplicity and clarity. Then, we have two options:
1. Measuring the log probability increases before and after the whole training session: As demonstrated in Figure 2, the loss of log probability due to forgetting starts immediately after the injection. Therefore, the log probability increases driven by FK injection will be underestimated, unless we inject FK only once or multiple times in a very short interval. Moreover, the amount of such underestimation will depend on the injection intervals, restricting the direct comparison of the measured values between different training setups.
2. Measuring the immediate log prob increases any time the model encounters the FK: As shown in Appendix H, the amount of the immediate increase is affected by the number of previous encounters, and it is especially high when the model has not encountered the knowledge before. Therefore, this option will compromise the interchangeability of effectivity between works experimenting with different numbers of injections.

There were similar considerations for the design of retainability (how much log probability decayed due to forgetting). In short, we measured the fraction of the log probability the model retained t steps after the final LAM, as the rate of forgetting may depend on the number of FK injections.

> Could you give an attempt already now?

We will make the following modifications to improve clarity and readability:
- First, as Reviewer 1qMb pointed out, we will change the symbol to represent effectivity (e.g., *Eff*), as the previous one ($\mathcal{E}$) can be misinterpreted as an error.
- Second, we will simplify the notations without changing their meaning. For example, we can simplify the notation of a model’s log probability on probe $q$ at timestep $t$ from $\ell(q,\theta_t)$ to $\ell(q,t)$.
- Third, we will elaborate on the intuitions behind the metric design and its meaning in more detail in the main text.

We hope this clarifies your follow-up questions. Thank you for taking the time to address this matter.
Thanks for the response. I want to apologize that the rating "poor" is simply a mistake. I changed it to "good" based on my overall assessment. Meanwhile, I have two specific questions for this paper. It would be appreciated if the authors could answer them. In general, I would like to keep my rating positive for this paper.
Thank you for taking time to respond to the weaknesses and questions. 

Your rebuttal, while generally clear, throws up a few more questions due to quite surface level answers.

> (1) we averaged all metrics across the entire dataset, and 

Averaging does not answer the question of how much overlap there is (only if it was a `perfectly' balanced dataset). Could you provide an analysis or study of overlap between pre-training data (PTD) and Fictional Knowledge (FK)? Could you then provide additional results on how metrics differ for these categories?

> (2) our primary focus is on relative changes in log probability. Thus, we don't anticipate a significant impact on our core findings and interpretations.

Similarly, here it would be very interesting to see how relative log probability changes behave under varying overlapping conditions. As other reviewers noted some information could be easily inferred by the LLM in general and therefore the change might be small for such knowledge (and the forgetting rate as well).

>  focus on designing the metrics was making them well-defined and easily adoptable in future works on analyzing fine-grained pretraining dynamics with different experimental setups

This needs further clarification and concrete examples and demonstrations.

>  We will strive to improve the clarity of our metrics in the revised version

Could you give an attempt already now?

Once again thank you for submitting the rebuttal, please elaborate on your answers and add more details. At this stage some things have become less clear.
Thanks for the clarifications. I am happy to increase my score by 1.