Instruction-tuned Language Models are Better Knowledge Learners (2024)

Abstract: In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe struggle to answer questions, even though the perplexity of documents is minimized. We found that QA pairs are generally straightforward, while documents are more complex, weaving many factual statements together in an intricate manner. Therefore, we hypothesize that it is beneficial to expose LLMs to QA pairs before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. Based on this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes on questions prior to training on documents. This contrasts with standard instruction-tuning, which learns how to extract knowledge after training on documents. Extensive experiments and ablation studies demonstrate that PIT significantly enhances the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8%.

heading: 1 Introduction
text: Large language models (LLMs) store vast amounts of factual knowledge in their parameters through large-scale pre-training, and this knowledge can be used to answer various questions such as “where is the world’s largest ice sheet located” (Brown et al., 2020; OpenAI, 2023; Chowdhery et al., 2022; Zhang et al., 2022; Touvron et al., 2023a,b; Gemini Team, 2023). However, this factual knowledge is static, meaning that it can become outdated as the
*Majority of the work done during an internship at Meta.
world evolves, or prove insufficient when LLMs are used in specialized or private domains.
To keep LLMs up-to-date, it is common to continue pre-training on new documents to store knowledge in parameters, which allows LLMs to effectively answer queries that require up-to-date information (Jang et al., 2022). A widely held view is that the factual knowledge stored in parameters can be elicited through prompting (Brown et al., 2020; Petroni et al., 2019; Roberts et al., 2020), and that instruction-tuning (also known as supervised finetuning or alignment) makes this elicitation more effective (Sanh et al., 2022; Wei et al., 2022; Ouyang et al., 2022). In the first part of this paper (§ 4), we conduct extensive experiments using Llama-2 (Touvron et al., 2023b) to answer the following question: to what extent can we augment the knowledge stored in modern LLMs by continued pre-training on new documents, either with or without subsequent instruction-tuning? We find that, as we train LLMs repeatedly over documents to the extent that perplexity is minimized to one, the percentage of questions regarding those documents that LLMs answer correctly increases consistently to 27.6%. Subsequent instruction-tuning further improves it to 30.3%, confirming that this widely used practice is useful to elicit more knowledge from LLMs.1 However, the amount of elicited knowledge is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the “perplexity curse”.2
In the second part of the paper (§ 5), we study methods to mitigate the perplexity curse by making LLMs more adept at absorbing knowledge from documents. Zhu and Li (2023a) presented an intriguing finding that training a randomly initialized
1This capacity might be underestimated by previous works due to using relatively small LMs or randomly initialized transformers, or lack of exhaustive training or instructiontuning (Wang et al., 2021; Hu et al., 2023; Zhu and Li, 2023a).
2Inspired by the “reversal curse” of Berglund et al. (2023).
5421
transformer from scratch on a mix of biographies and related questions resulted in strong generalization to new questions. However, understanding the reasons behind this finding and exploring ways to practically apply it for absorbing knowledge from new documents requires further investigation. We found that question-answer (QA) pairs are generally straightforward and easily digestible, while documents tend to be more complex and cluttered, often weaving many factual statements together in a more intricate manner. Therefore, we hypothesize that it is beneficial to deliberately expose LLMs to QA data before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. We refer to this as pre-instruction-tuning (PIT) and conduct comprehensive experiments to benchmark different variations of this method. As shown in Fig. 1, our best-performing variation starts with training exclusively on QA pairs (e.g., “who handled the editing of Oppenheimer”) to grasp how knowledge is accessed. This is followed by training on a combination of these QA pairs and associated documents (e.g., “who handled the editing of Oppenheimer” and a document about “Oppenheimer”). In this phase, LLMs enhance their ability to absorb knowledge from information-dense documents, building upon the QA pairs that they have already mastered. To study continual knowledge acquisition, we build a dataset named Wiki2023, which includes a collection of documents from Wikipedia that are relevant to the year 2023. Comprehensive experiments on Wiki2023 demonstrate that after PIT, LLMs exhibit an enhanced ability to absorb knowledge from new documents (e.g., a document about “Barbie”). Detailed ablation studies reveal that this ability primarily stems from prioritizing learning how to access knowledge over learning to encode knowledge from documents. Overall, PIT significantly outperforms the standard instruction-tuning approach (§ 5.1 and § 5.2), improving QA accuracies by 17.8% on Llama-2 7B (30.3% 48.1%) and 16.3% on Llama-2 70B (46.4% 62.7%). Moreover, PIT also enhances the ability to absorb knowledge from documents of a different domain, shedding light on the potential to scale this method up to a wider variety of documents and instructions for more robust generalization (§ 5.4).

heading: 2 Building a Dataset to Study Continual Knowledge Acquisition
text: To assess the ability of LLMs to learn knowledge from new documents, it is essential to use a document corpus with minimal overlap with the original pre-training corpus. This ensures that when an LLM correctly answers questions, we can confidently attribute this capability to its learning from the new documents, rather than encountering similar questions in its original pre-training corpus. In this section, we describe a methodology for building such a corpus from Wikipedia.

heading: 2.1 Wiki2023 Document Corpus
text: In the following experiments (§ 4 and § 5), we use Llama-2 (7B and 70B) (Touvron et al., 2023b) since it is one of the best-performing LLMs. We use Wikipedia articles classified under the “2023” Category including topics from diverse domains such as films, arts, economics, politics, events, etc.3 The likelihood that this factual information is not included in the original training corpus is supported by the low QA performance in Tab. 1 (9.5%/17.2%
3https://en.wikipedia.org/wiki/Category:2023
for 7B/70B).4 To accelerate the training process, we only use the first section of each article, which offers a thorough summary and contains many factual statements. The number of collected documents and an example document about “Oppenheimer” can be found in Fig. 2 and Fig. 3. We refer to this as the Wiki2023 dataset.

heading: 2.2 Wiki2023 Question-answer Pairs
text: To collect QA pairs for either instruction-tuning or performance evaluation, we employ publicly available LLMs to generate diverse questions and their respective answers given the article as context, following the Prompt 1 in Appendix A. On average, 4.93 questions are generated for each article. Fig. 2 and Fig. 3 show the detailed statistics and example QA pairs about “Oppenheimer”, respectively.

heading: 2.3 Splits
text: Among all domains, we select the film domain for evaluation and randomly select 256 articles as the test split (Wiki2023-film-test). We continually train LLMs on documents from the test split (Wiki2023-film-test-doc), and assess their performance based on the accuracy of corresponding questions (Wiki2023-film-test-QA). The remaining 1720 articles and corresponding QA pairs (Wiki2023-film-train) will be used to study dif-
4It is important to note the difficulty in completely avoiding factual overlap between Wiki2023 and the pre-training corpus of Llama-2. For example, a film released in 2023 might have had information available before 2023. Data duplication detection is an active research direction, which falls beyond the focus of this study.
ferent training strategies, which corresponds to the in-domain setting in Fig. 2. We also train on other domains before evaluation on the film domain to study the effectiveness of different methods across domains, which corresponds to the cross-domain setting in Fig. 2.

heading: 3 Experimental Settings

heading: 3.1 Objectives
text: When training on documents, we prepend a <bos> token and compute the standard next-token prediction loss by averaging over all tokens in the document: Ld = − ∑ t logP (dt|d<t)/|d|.5 When training on QA pairs, we compute the average negative log-likelihood loss only on tokens in the answer given the question as the prefix: La = −∑t logP (at|q,a<t)/|a|. Fig. 3 presents an example document alongside QA pairs, where tokens used for computing losses are highlighted.

heading: 3.2 Hyperparameters
text: When pre-training on documents, we use a batch size of 256 documents and an initial learning rate of 3e-5. During instruction-tuning on QA pairs, we use the same batch size of 256 QA pairs but opt for a reduced initial learning rate of 5e-6 because the number of tokens in a single batch is lower. Details can be found in Appendix B.

heading: 3.3 Evaluation Metrics
text: Since most answers are relatively short, we use exact match (EM) as our primary metric
5We do not append a <eos> token at the end of documents because we only use the first section, which does not signify the conclusion of the entire article.
(Kwiatkowski et al., 2019). To assess longer responses and accommodate minor lexical differences, we also report answer recall and ROUGE-L. Details can be found in Appendix C.

heading: 4 How Much Knowledge Can LLMs Absorb via Continued Pre-training Followed by Instruction-tuning?
text: Factual knowledge stored in the parameters of LLMs can be accessed and applied to answering questions through prompting without additional training (Brown et al., 2020; Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020). With additional instruction-tuning (also known as supervised fine-tuning) on high-quality data (Sanh et al., 2022; Wei et al., 2022), knowledge seems to be more effectively elicited from LLMs. However, when LLMs correctly answer a question, the source of the knowledge is unclear due to the diversity of the pre-training data. For instance, when answering the question “where is the world’s largest ice sheet located”, do LLMs derive their response by recalling and generalizing information from a seen document about the Antarctic ice sheet, or do they merely repeat answers from similar questions encountered in the training data? This distinction is crucial, as the former scenario implies an ability to comprehend documents and effectively store knowledge within parameters in a way that can be elicited later, whereas the latter is mere rote memorization.
Several works have studied this problem and the predominant finding is that LMs struggle to answer questions about documents they have been trained on (Wang et al., 2021; Zhu and Li, 2023a). It is important to note, however, that these experiments were mainly conducted using relatively small LMs
such as BART, T5, or GPT-2 (Wang et al., 2021; Jang et al., 2022; Hu et al., 2023), using randomly initialized transformers (Zhu and Li, 2023a), or without instruction-tuning (Ovadia et al., 2023). This makes us wonder what are the actual limits of modern LLMs to absorb knowledge from new documents and answer questions about them using the standard continued pre-training followed by instruction-tuning recipe. In this section, we run extensive experiments using Llama-2 7B and 70B on Wiki2023-film to test their limits.

heading: 4.1 Vanilla Continued Pre-training and Instruction-tuning
text: Experimental settings We experiment with two standard settings and assess their performance by answering associated questions.
• Continued pre-training: train on test documents without instruction-tuning (Fig. 4 ➀).6
• Standard instruction-tuning: train on both train and test documents before instruction-tuning on train QA pairs (Fig. 4 ➁).
We perform instruction-tuning for a single epoch since more epochs usually result in diminished performance. For training on documents, we opt for multiple epochs (10/5 for a 7B/70B model), which allows for effective knowledge acquisition and remains affordable for corpora of moderate sizes.
Experimental results As shown in Tab. 1, the relatively low performance of the original Llama2 model (9.5%/17.2% for 7B/70B) indicates that
6We found that LLMs struggle to adhere to the QA format after training on raw documents for multiple epochs. Therefore, we include a small set of QA pairs (64) during continued pre-training to prevent LLMs from forgetting the QA format.
1.0
2.0
3.0
4.0
0
5
10
15
20
25
30
0 10 20 30 40 50 #epochs
10
20
30
40
QA acc. after cont. pre-training (%) QA acc. after cont. pre-training & instruction-tuning (%) Perplexity on the documents Knowledge retention accuracy (%)
9.5
20.3 27.6
30.6 31.5 33.0 33.3
23.1
30.3 32.8 34.5
36.1 37.6
minimized perplexity
(a) Training dynamics w/ (Fig. 4 ➁) and w/o instruction-tuning (Fig. 4 ➀). Reduction in perplexity consistently leads to improvement in QA accuracy, indicating that factual knowledge acquisition necessitates exhaustive loss minimization.
1.0
2.0
3.0
4.0
0
5
10
15
20
25
30
0 10 20 30 40 50 #epochs
10
20
30
40
QA accuracy (%) lr=1e-5 Perplexity on the docs lr=1e-5 Knowledge retention (%) lr=1e-5
3e-5 3e-5 3e-5
5e-5 5e-5 5e-5
21.3 25.2 26.3
31.5 33.0 33.3
27.8 30.9 31.3
minimized perplexity
(b) Training dynamics with different learning rates (Fig. 4 ➀). After perplexity is minimized, larger learning rates usually lead to less overfitting to deceptive patterns in documents and better generalization when responding to questions.
most knowledge in the test documents is not included in the original pre-training corpus. After continued pre-training on documents, performances increase to 27.2%/41.7%, indicating that LLMs can absorb some amount of knowledge. Instruction-tuning further increases the performance to 30.3%/46.4%, confirming the effectiveness of this standard recipe. This observation is different from Zhu and Li (2023a), which demonstrates that instruction-tuning after pre-training is ineffective on a randomly initialized GPT-2-like transformer. The difference probably arises because Llama-2, through its pre-training on diverse corpora comprising raw documents and QA data, has developed a certain degree of proficiency in extracting knowledge from its parameters via questions. We also report the performance where the corresponding document is directly provided to Llama-2 as context (“open-book w/ doc” in Tab. 1). The significant gap between closed-book and openbook settings suggests that retrieving knowledge from the parameters of LLMs is still challenging.

heading: 4.2 Analyzing the Training Dynamics: Perplexity and Generalization
text: How does lower perplexity of documents lead to generalization to answering related questions? We vary the number of epochs (Fig. 5(a)) and learning rate (Fig. 5(b)) for continued pre-training on documents and monitor three metrics to study the
training dynamics.7
• Knowledge acquisition QA accuracies on test questions measured by exact match.
• Perplexity of documents We compute perplexity (PPL) on all tokens within the documents.
• Knowledge retention We approximate the retention of accumulated knowledge during pretraining by assessing the QA accuracy on the Natural Questions (NQ) dataset. NQ was released in 2019, and primarily includes questions based on Wikipedia articles from that time.
Experiment results
• As shown in Fig. 5(a), QA accuracy consistently improves as perplexity approaches one, indicating that factual knowledge learning necessitates exhaustive loss minimization over all tokens. This contrasts with learning general skills, where overly optimizing leads to overfitting.
• As shown in Fig. 5(a) and Fig. 5(b), among all cases where LLMs have minimized perplexity on documents, for reasonably small learning rates (5e-5 is too large and leads to overfitting), cases trained with more epochs or larger learning rates typically exhibit superior QA performance. We
7Since we always decay the learning rate to 10% of its initial value, training for more epochs is not the same as continuing training from a checkpoint obtained after fewer epochs.
hypothesize that more aggressive training leads to less overfitting to deceptive patterns in documents and better generalization when responding to questions.
In summary, lower perplexity does lead to stronger generalization when responding to questions, but it comes at the expense of forgetting previously acquired knowledge.

heading: 5 Improving LLMs in Absorbing Knowledge from Documents
text: The amount of knowledge elicited through the standard instruction-tuning is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the “perplexity curse”. Our next question is how can we improve the ability of LLMs to absorb knowledge from documents to mitigate the perplexity curse. The main challenge is the gap between the way knowledge is presented in raw documents and how it is accessed through question-answering. We found that QA pairs are generally straightforward, while documents tend to be more complex and cluttered, weaving many factual statements together in a more intricate manner. Using Fig. 3 as an example, the answer to the question “who handled the editing of Oppenheimer” is included in a sentence in the middle of the article “Editing was handled by Jennifer Lame ...”, which does not explicitly mention “Oppenheimer”. During training, LLMs must understand the context and deduce that “editing” refers to “the editing of Oppenheimer” to effectively encode this knowledge in the parameters.
Zhu and Li (2023a) studied this problem by training a randomly initialized GPT-2-like transformer from scratch on synthetic biographies and evalu-
ated its ability to answer questions about the individuals. They found that training on a mix of biographies and questions related to half of those biographies led to strong generalization when answering questions about the remaining half of biographies, which resembles setting ➃ in Fig. 4. In contrast, training on biographies and QA pairs sequentially failed. However, the key contributor to the success remains uncertain because the data were blended together, and it is unclear how to apply this practically to absorb knowledge from new documents. Inspired by our observation of the different difficulty levels between QA pairs and documents, and the finding from Zhu and Li (2023a), we hypothesize that it is beneficial to deliberately expose LLMs to instruction-tuning data before continued pre-training so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed. We refer to this as pre-instruction-tuning (PIT) and study various implementations of PIT prior to continued learning (§ 5.1), followed by detailed ablations identifying the keys contributor to performance (§ 5.2 and § 5.3), and finally assess how well PIT performs across domains (§ 5.4). We adhere to the hyperparameters outlined in § 3.2 and perform PIT for 3 epochs unless specified otherwise.

heading: 5.1 Variants of Pre-instruction-tuning
text: Pre-instruction-tuning w/ QA only We start with exposing instruction-tuning data before continued pre-training on documents—training on topically related QA pairs before training on test documents (Fig. 4 ➄). This can be directly compared with the continued pre-training setting (Fig. 4 ➀). The intuition is that questions help LLMs recognize key types of information, enabling LLMs to focus on important information during pre-training on subsequent documents, even though the questions are not directly tied to the documents. For example, training on a question like “who handled the editing of Oppenheimer” could help LLMs pay attention to screenwriters when training on new documents like “Barbie”. As shown in Tab. 1, this method outperforms continued pre-training, especially on larger LLMs (27.6%/41.7% 28.6%/49.7% for 7B/70B). The ablation that trains on QA data after training on documents (“instruction-tuning w/o train doc” in Tab. 2) is ineffective, confirming the importance of training on questions as a warm-up before encoding documents.
Setting names Setting configurations EM Rec. R-L
baselines continued pre-training ➀ test doc 27.6 31.6 43.8
+instruction-tuning ➁ train doc + test doc train QA 30.3 34.7 47.4 +instruction-tuning (w/o forget) ➂ train doc + test doc train QA + test doc 30.2 34.1 46.4 +instruction-tuning (w/o train doc) test doc train QA 27.1 30.7 42.3 weighted continued pre-training test doc (weighted) 27.7 32.7 43.3 adapted continued pre-training train doc test doc 26.9 32.7 44.2 mix all data ➃ train QA + train doc + test doc 39.4 44.6 56.7
Pre-instruction-tuning on QA and documents sequentially Our second implementation trains on QA and associated documents sequentially (Fig. 4 ➅), with the intuition that the ability to absorb knowledge from documents can be strengthened if an LLM is trained on the complex documents after it has grasped the associated simpler QA pairs. For instance, if an LLM has already learned that “Jennifer Lame” is the answer to “who handled the editing of Oppenheimer”, training on the document “Editing was handled by Jennifer Lame” can more efficiently refine its storage of knowledge in its parameters. As shown in Tab. 1, PIT on QA pairs and documents sequentially surpasses the QAonly variant (Fig. 4 ➄) and standard instructiontuning (Fig. 4 ➁) (30.3%/46.4% 32.5%/54.6% for 7B/70B), demonstrating its effectiveness.
Pre-instruction-tuning The effectiveness of PIT depends on ensuring that the associated QA pairs are already learned before encoding the respective documents. However, we observed that after training on documents (train doc in Fig. 4 ➅), the accuracy for corresponding questions (train QA in Fig. 4 ➅) dropped from almost perfect to 30%, indicating severe forgetting. To fix this, we train on the associated QA pairs and documents together (Fig. 4 ➆). As shown in Tab. 1, this significantly improves the performance, outperforming all other approaches, including mixing all data together (Fig. 4 ➃), by
a large margin (39.4%/57.1% 45.5%/62.7% for 7B/70B). Training on both QA pairs and documents prevents forgetting, but it also obscures how the learning process works. It is unclear whether LLMs grasp QA pairs before encoding knowledge from documents, or if it works the other way around. In the following section, we deliberately arrange the order of QA pairs and documents during training to examine this, which leads us to propose an improved version of PIT.

heading: 5.2 Pre-instruction-tuning++
text: We first study how the performance varies with different numbers of epochs. As shown in Tab. 2, training for 1 epoch is insufficient, and the performance of 3, 5, or 10 epochs is similar. We fix the number of epochs to 3 and arrange the order of QA pairs and corresponding documents as shown in Fig. 6 in Appendix D. The interleaved arrangement cycles through all the data 3 times, ensuring that in each epoch, questions either precede or follow their associated documents. On the other hand, the grouped arrangement clusters each example’s 3 appearances together, guaranteeing that the repeated questions are positioned either before or after their respective repeated documents. As shown in Tab. 2, positioning QA pairs before corresponding documents achieves better performance in both grouped and interleaved arrangements, indicating that during PIT, the learning mechanism prioritizes under-
standing how to access knowledge before learning to absorb information from the more complex and information-dense documents.
Based on this, we propose an improved variant called pre-instruction-tuning++, which trains exclusively on QA pairs to understand patterns of knowledge access, then progresses to training on a combination of QA and document data to align knowledge access through questions and knowledge encoding from documents (Fig. 4 ➇). As shown in Tab. 2, PIT++ significantly outperforms PIT (Fig. 4 ➆) from 45.4% to 48.1%, while training on QA data after on the mix (PIT-- in Tab. 2) does not yield additional benefits. This reinforces our hypothesis that understanding how knowledge is accessed aids in absorbing knowledge from documents, and therefore, should be prioritized.

heading: 5.3 Ablation Studies
text: Standard instruction-tuning is inferior not due to forgetting A drawback of standard instructiontuning is that knowledge in test documents might be forgotten after training on QA pairs (a phenomenon also known as the “alignment tax” (Ouyang et al., 2022)). To show that the lower performance of standard instruction-tuning is not due to forgetting, we add a setting where we mix train QA with test documents during instruction-tuning to prevent forgetting (Fig. 4 ➂). As shown in Tab. 2, this does not help, confirming our hypothesis.
Pre-instruction-tuning is not simply upweighting salient tokens from documents We include an ablation inspired by Hu et al. (2023) which upweights tokens when pre-training on documents to focus on salient information. We assign a weight of 1.0 to tokens in documents that are included in the answers (e.g., “Jennifer Lame” in the sentence “Editing was handled by Jennifer Lame”), and assign a lower weight of 0.5 to other tokens. As shown in Tab. 2, this weighted continued pretraining is ineffective, confirming our hypothesis.

heading: 5.4 Cross-domain Generalization
text: We validated the effectiveness of PIT by training and evaluation on data from the same domain (Wiki2023-film). Can PIT make LLMs better at absorbing knowledge from documents of a different domain? To this end, we follow the cross-domain setting outlined in Fig. 2—training on other domains (Wiki2023-other-train) and testing on the film domain (Wiki2023-film-test). The results of standard instruction-tuning and PIT, in both in-domain and cross-domain settings, are detailed in Tab. 3. Even though it is not as effective as the indomain counterparts, cross-domain PIT still significantly outperforms instruction-tuning, demonstrating that it can generalize across different domains. This finding sheds light on the potential to scale this method up to a broader range of documents and instructions for more robust generalization.
We also evaluate the effectiveness of PIT in two other scenarios: (1) when applied to non-Wikipedia documents, and (2) when addressing questions asked by real users. For the first scenario, we take the Llama-2 7B model trained with PIT on 2023Wiki-other and further train it on biographies synthesized in Zhu and Li (2023a) (bioS). Then, we evaluate based on questions about the individuals. For the second scenario, we manually search Google using questions generated by LLMs from Wiki2023-film-test, collect a total of 93 similar questions from real users by leveraging Google’s “People Also Ask” feature, and then evaluate Llama-2 7B on these questions. As shown in Tab. 4, PIT outperforms baselines in both scenarios, demonstrating its generalization ability.

heading: 6 Related Work

heading: 6.1 Continual Knowledge Acquisition
text: Several works have studied whether LMs can answer questions about information in documents they have been trained on. Wang et al. (2021); Jang et al. (2022); Hu et al. (2023) use relatively
small LMs such as BART (Lewis et al., 2020a), T5 (Raffel et al., 2020), or GPT-2 (Radford et al., 2019). Ovadia et al. (2023) focus on the comparison between RAG and continued pre-training approaches without using instruction-tuning. Zhu and Li (2023a,b) examine this problem from a similar angle as ours using a GPT-2-like transformer trained from scratch on synthetic biographies and fine-tuned on QA pairs related to the individuals. They examined a mixed training setting on both biographies and QA pairs, which is our major motivation to study different strategies to incorporate QA data before continued pre-training. Other works study adapting LLMs to new domains via various strategies (Zhang et al., 2023; Cheng et al., 2023; Han et al., 2023; Wu et al., 2023; Nguyen et al., 2023; Zhao et al., 2023).

heading: 6.2 Instruction-tuning or Alignment
text: Instruction-tuning (also known as supervised finetuning) on high-quality annotated data (Sanh et al., 2022; Wei et al., 2022; Mishra et al., 2022; Iyer et al., 2022; Kopf et al., 2023; Zhou et al., 2023; Sun et al., 2023b,a) and/or data generated by proprietary models (Taori et al., 2023; Chiang et al., 2023; Wang et al., 2023b; Ivison et al., 2023), or alignment with reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) (Ouyang et al., 2022; Touvron et al., 2023b; Rafailov et al., 2023; Tian et al., 2023) has been a central topic recently because it elicits knowledge from LLMs and enhances various abilities to handle questions from users. We focus on factuality and study the best way to perform instructiontuning to elicit factual knowledge from LLMs.

heading: 6.3 Analyzing the Training Dynamics of LMs
text: Many works study the training dynamics of LMs from different perspectives. Carlini et al. (2022) quantifies memorization across model sizes and the frequency of data duplication. Tirumala et al. (2022) finds that larger LMs memorize training data faster with less overfitting. Xia et al. (2023) shows that perplexity is more predictive of model behaviors than other factors. Dery et al. (2022) studies end-task aware pre-training using classification tasks and RoBERTa models. Jia et al. (2022) adds a pre-training objective to encourage the vector for each phrase to have high similarity with the vectors for all questions it answers. Our work differs in that we specifically focus on the capacity of recalling and generalizing information from a seen document to answer questions.

heading: 6.4 Retrieval-augmented Generation
text: Retrieval-augmented generation (RAG) is a widely used approach to incorporate new knowledge into LLMs by augmenting fixed LLMs with retrieved information from external sources (Chen et al., 2017; Guu et al., 2020; Lewis et al., 2020b; Borgeaud et al., 2022; Wang et al., 2023a; Alon et al., 2022; He et al., 2021; Sachan et al., 2021; Izacard et al., 2023; Lee et al., 2022; Jiang et al., 2022; Shi et al., 2023; Jiang et al., 2023; Asai et al., 2023; Nakano et al., 2021; Qin et al., 2023; Lin et al., 2023). While RAG is effective in reducing hallucinations commonly experienced when relying solely on knowledge stored in parameters, its retrieval and generation process adds extra latency and complexity. In contrast, continued pre-training to store knowledge in parameters and utilizing the stored knowledge to answer questions in a closedbook manner are simpler and faster at inference time. Enhancing this capability is also scientifically significant, as it represents a fundamental step in employing LLMs as dependable assistants for accessing information. Therefore, this paper focuses on exploring parametric approaches.

heading: 7 Conclusion
text: We study the best way of continued training on new documents with the goal of later eliciting factual knowledge and propose pre-instruction-tuning that learns how knowledge is accessed via QA pairs prior to encoding knowledge from documents. Extensive experiments and ablation studies demonstrate the superiority of pre-instruction-tuning versus standard instruction-tuning. Future directions include scaling this method up to a broader range of documents and instructions for more robust generalization.

heading: 8 Limitations
text: The Wiki2023 dataset provides a relatively clean testbed for studying continual knowledge acquisition. However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning on QA data in this paper. The effectiveness of preinstruction-tuning with different types of data for enhancing other skills like reasoning or comprehension is something that needs to be explored in future studies.

heading: Acknowledgements
text: We would like to thank Zeyuan Allen-Zhu, Zexuan Zhong, Shuyan Zhou, Frank F. Xu, Qian Liu, and Ruohong Zhang for their help with the experiments and constructive feedback.

heading: B Hyperparameters
text: We use AdamW (Loshchilov and Hutter, 2019) with β1 = 0.9, β2 = 0.95, and a weight decay of 0.1. We decay the learning rate to 10% of its initial value using a cosine scheduler without warm-up. When pre-training on documents, we use a batch size of 256 documents and an initial learning rate of 3e-5. During instruction-tuning on QA pairs, we use the same batch size of 256 QA pairs, but opt for a reduced initial learning rate of 5e-6 because the number of tokens in a single batch used for computing losses is lower. The number of epochs varies depending on the setting and is detailed in the corresponding sections.

heading: C Evaluation Metrics
text: At inference time, we use greedy decoding to generate answers given questions as context, following the format in Fig. 3. To evaluate the original Llama2, we add 5 QA pairs as in-context exemplars to make sure it follows the QA format. Since most questions are simple factoid questions and most answers are relatively short, we use exact match (EM) as our primary metric (Kwiatkowski et al., 2019), which measures whether the model’s output matches the gold answer exactly after normalization (e.g., remove articles and punctuations). To assess longer responses and accommodate minor lexical differences, we also report answer recall, which

heading: QA before documents (grouped)

heading: QA after documents (grouped)
text: assesses if the gold answer appears in the model’s output, and ROUGE-L, which measures the longest common subsequence between the model’s output and the gold answer.

heading: D Details of Ablation Studies
text: We arrange the order of QA pairs and corresponding documents as shown in Fig. 6 to study the learning mechanism of pre-instruction-tuning.

heading: Future Work
text: The effectiveness of preinstruction-tuning with different types of data for enhancing other skills like reasoning or comprehension is something that needs to be explored in future studies