Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning
Authors: Yujian Liu
Shiyu Chang
Tommi Jaakkola
Yang Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Hallucinations
instruction-tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Recent studies have identified one aggravating factor of LLM hallucinations as the knowledge inconsistency between pre-training and fine-tuning, where unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs. In this paper, we propose a novel fine-tuning strategy called Prereq-Tune to address this knowledge inconsistency and reduce hallucinations. Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge, so the model learns only the task skills without being impacted by the knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional prerequisite learning stage to learn the necessary knowledge for SFT, allowing subsequent SFT to focus only on task skills. Prereq-Tune can also be combined with fictitious synthetic data to enhance the grounding of LLM outputs to their internal knowledge. Experiments show that Prereq-Tune outperforms existing baselines in improving LLM's factuality across short QA and long-form generation tasks. It also opens new possibilities for knowledge-controlled generation in LLMs. Our code is available at https://github.com/UCSB-NLP-Chang/Prereq_tune.git.

Published as a conference paper at ICLR 2025
FICTITIOUS SYNTHETIC DATA CAN IMPROVE LLM FACTUALITY VIA PREREQUISITE LEARNING
Yujian Liu
Shiyu Chang
UC Santa Barbara
UC Santa Barbara chang8 70 edu
Tommi Jaakkola MIT CSAIL
Yang Zhang
MIT-IBM Watson Al Lab
tommi@csail mit edu
yang. . zhang2 ibm com.
ABSTRACT
Recent studies have identified one aggravating factor of L.LM hallucinations as the knowledge inconsistency between pre-training and fine-tuning, where unfa- miliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs. In this paper, propose novel fine-tuning strategy called PREREQ-TUNE address this knowledge inconsistency and reduce hallucinations. Fundamentally, PREREQ-TUSE -UNE disentangles the learning of skills and knowledge, the model learns only the task skills without being impacted by the knowledge inconsistency. additional prerequisite learning
To achieve this, PREREQ- TUNE introduces stage to learn the necessary knowledge for SFT. allowing subsequent SFT to focus only on task skills. PREREQ- TUNE can also be combined with fictitious synthetic data to enhance the grounding of LLM outputs to their internal knowledge. Ex- periments show that PREREQ-TUE TUNE outperforms existing baselines in improving LLM's factuality across short QA and long-form generation tasks. It also opens new possibilities for knowledge-conttolled generation LLMs. Our code is avail- able at atlhttas://gtthhie com/ UCSB NLP Chang tune .it
INTRODUCTION
Hallucination large language models (LLMs) refers to the phenomenon where LLMs' outputs look plausible but diverge from real-world facts. It has become major concern of LLMs, seriously undermining their reliability and trustworthiness (Huang et al. 2023 Ji et al. 2023).
Recent research has unveiled one aggravating factor of LLM hallucination, which is the knowledge inconsistency between the pre- are-training and tuning (e.g., instruction or fine-tuning) stages (Gekhman et al.
2024 Kang al. 2024 Lin al. 2024). More specifically, if the tuning stage involves train- ing examples that require knowledge that an L.M has not seen during pre-training, then the LLM would be misled fabricate plausible but wrong answers unfamiliar questions (Schulman 2023
Gao 2021; Goldberg (203).. For example, consider fine-tuning model for question answering (QA) task with the example 'When was John Estes born?' and assume that the LLM has never learned about John Estes during pre-training. However, since the LLM is still trained to produce the correct answer, '1987', it consequently encouraged respond with a random legitimate year whenever it is asked about the birth year of any unknown person, thus giving rise to hallucination.
These findings highlight an important but previously understudied consideration of L.M training, which is the disentanglement between knowledge and skill. Specifically, the abovementioned knowl- edge inconsistency issue indicates that knowledge can interfere with the learning of skills during fine-tuning. If piece of knowledge required in fine-tuning example is non-existent in the LLM, training on this example will not teach the LLM the desired skills (e.g., answering questions based on its internal knowledge); instead, the training will incentivize the LLM to fabricate non-existent facts that look plausible. Thus, only by disentangling knowledge from skill during the tuning stage remove such interference.

---

Published as a conference paper at ICLR 2025
Stop I: Precequisite Loaming
Step Supervised Fine- Time-Tuning
Step 3 Inforanc
Fads John Estes
When was] John Estes born?
Q: When Alan Turing bomt?
A: 1987
Contimonus pre-trainingg
Supervised fine tuning
Genontion
skill LoRA
knowllokge
Base
LoRA
LLM
LoRA
LLM
knowledgge
LIM
LoRA
Figure
Overview the proposed PRERBQ-TURE TUNE strategy.
An ideal powerful disentangling mechanism should achieve maximal robustness against the knowl- edge inconsistency, to the extent that even fictitions synthetic data, which has zero knowledge over- lap with pre-training, would still warrant successful fine-tuning without encouraging hallucination. Unfortunately, this may sound too good to be true, because such an effective disentanglement strat- egy has yet to be developed for LLM tuning.
In this paper, we propose PREREO TREREO-TUNE, an innovative LLM fine-tuning strategy that explicitly resolves the disentanglement challenge, and therefore can effectively reduce LLM hallucinations. As shown in Figure PREREO-TUNE consists of two stages, an innovative prerequisite learning stage and a supervised fine-tuning (SFT) stage. In the prerequisite learning stage, we train a LoRA (Hu al.) 2022), called the knowledge LoRA, that learns all the necessary knowledge required for SFT. In the SFT stage, the knowledge LoRA is frozen, and new LoRA, called the skill LoRA, is imposed top of the knowledge LoRA, and is trained perform the SFT task. Since the knowledge LoRA already eliminates the knowledge mismatch issue, the skill LoRA can focus on learning the skills without inducing hallucinations.
With the prerequisite learning as a disentanglement mechanism, PREREQ- TUNE can turn fictitious synthetic data, which is otherwise detrimental LLM factuality, into powerful weapon further reduce LLM hallucinations. In particular, using fictitious synthetic data, we can create multiple knowledge LoRAs that contain different versions of knowledge about the same fictitious entity, and then teach the skill LoRA to produce different answers based on which knowledge LoRA is in use. In this way, we can force the skill LoRA ground the LLM's answers its internal knowledge, thus reducing hallucinations. Moreover, unlike real data which are expensive to obtain and label, fictitious synthetic data can cheaply scaled up, which would further enhance the LLM's factuality fine-tuning paradigm.
thanks to PREREQ CEEREQ-TTNE, informing
Our experiments reveal that PREREQ-TE TUNE can teach an LLM to not only ground its answer to the knowledge LoRAs, but also, more surprisingly, generalize the grounding the LLM original pre-trained knowledge when the knowledge LoRA removed. As a result, PREREQ- TUNE can significantly outperform the existing -fate-of-the-art hallucination reduction algorithms in improv- ing LLM's factuality across short QA and long-form generation tasks. addition, PREREQ -UNN enables a more modular design of LLM, with plug-and-play knowledge modules that control the knowledge access and skill module that works generically with any knowledge sources, which opens up many new possibilities far beyond hallucination reduction, such as novel retrieval aug- mented generation (RAG) paradigms, privacy protection, etc.
2 RELATED WORKS
Reducing LLM hallucinations. Numerous studies have sought to detect and mitigate hallucinations
in LLMS [Weng] 2024 Li et al.) 2023a Chern et al.) 2023 Azaria & Mitchell] 2023 Manakul
et al. 2023 Chen et al. 2024 Hou et al. 2024 A common approach to reduce hallucinations is
leveraging supporting evidence, either retrieved from external knowledge sources or generated by
LLMs. Such evidence is often provided
input to help LLMs generate more factual responses
(Shuster et al. 2021; Nakano et al., 2022 Menick et al.) 2022 Sun et al. 2023b Asai et al. 2024]
Feng et al. 2024); they assist in detecting incorrect statements and guide LLMSs to post-edit
2

---

Published as a conference paper at ICLR 2025
2024 Lei et al. 2023 Mishra et al. 2024). Beyond using additional evidence, several works have proposed decoding a algorithms to improve LLM factuality during inference (Lee et al. 2022
their own generations (Gao et al. 2023 Dhuliawala et al.
Chuang et al. 2024 Li et al. 2023b).
More recently, some studies have also explored answer abstention when LLMs encounter unfamiliar
questions (Zhang al. 2024 Yadkori al., 2024 Yang et al. [023 Cheng et al.
(024). Our work
aligns with recent efforts that fine-tune LLMs so that they inherently generate less hallucinated
contents (Tian et al., 2024 Lin al., 2024; Kang et al. 2024; Ghosal al., 2024). Unlike existing
methods, we focus on the dusentanglement of skills and knowledge and the groundedness of LLMs' outputs, which leads to superior performance hallucination reduction.
Fine-tuning LLMs with synthetic data. Synthetic data has shown great potential in fine-tuning L.MMs, offering scalability by automatically creating instruction-respons pairs with minimal human supervision (Wang et al. 2023: Li et al. 2024] Gupta et al. 2024 Yu et al. 2024 Haluptzok et al.) 2023). Additionally, the generation process can be designed to control the training data, allowing models to be trained for specific skills or aligned with particular human values (Sudalairaj et al. 2024 Sun et al. 2023a 2024; Kaur et al. 2024). However, most works do not consider factuality in fine-tuning. The work closest to ours Jones et al. (2024), which fine-tunes LLMs synthetic tasks to reduce hallucinations. However, they focus on improving consistency with evidence provided in the context, whereas improve LLMs' inherent factuality without provided evidence.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
Given a pre-trained LLM (without instruction-tuning parameterized by Po, a downstream task T, and its corresponding dataset Dy, our task is to fine-tune the LLM to the downstream task. For concreteness, we will base our illustration of the algorithm on a specific task, biography generation. Generalization to other tasks is discussed in Section 3.6 In this task, D7 would typically contain requests to generate a biography of a certain person, e.g., 'Generate a biography for John Estes', as well as the corresponding human- human-written biographies.
We will focus on two aspects embedded in Dr - the knowledge and skill. The knowledge refers to all factual information involved in D7, and the skill describes the LLM's ability to output the desired format to different user queries, e.g., the ability to answer a question or write biographies in the desired formats. It is worth mentioning that the terms we use here have also been discussed in existing literature. In particular, Zhou et al. (2023) uses the term 'style and format' to describe the LLM's ability output the desired format, which corresponds to the 'skill' in our paper.
The challenge we aim to resolve is that Dy may require knowledge that the pre-trained LLM 6s does not know, which would encourage LLM hallucination. Specifically, although D, involves both knowledge and skill, the difficulty of learning the two is different, and evidence has shown that knowledge and skills are acquired at differrent stages of the fine-tuning process skills are generally acquired earlier than knowledge (Gekhman et al. 2024). It is believed that such gap (in difficulty and learning stages) between knowledge and skill acquisition since skills are learned before the new knowledge is absorbed, LLMs are misled into fabricating non-existent facts whenever they see unfamiliar questions. as a shortcut to minimize fine-tuning loss (Gekhman et al.] 2024) Kang et al. 2024] [Li et al., 2024). Our goal is to design fine- tuning strategy that can learn the skill, e.g., writing biographies, without being impacted by the inconsistency between the knowledge involved in D7 and the pre- rre-triind knowledge.
likely cause of hallucination
3.2 BASIC PREREQ-TURE STRATEGY
To achieve the disentanglement between knowledge and skill, the core idea of PREREQ-TUNE is to introduce a prerequisite learning stage that equips the LLM with the necessary knowledge required for subsequent SFT. More specifically, given the task dataset, Dy, PREREQ-TUNE introduces prerequisite knowledge dataset, Dmow, that contains all the prerequisite knowledge for the questions in Dr. For biography generation, the prerequisite knowledge includes all the knowledge about the target persons that is covered in the biographies in Dy, such as birth year, birthplace, representative works, etc. Methods of creating the prerequisite knowledge dataset detailed in Section 3.4
3

---

Published as a conference paper at ICLR 2025
With the dataset pair, (Dinow, Dr), the training process of PREREQ- TUNE consists of the following two steps, as illustrated in Figure I
Step 1: Prerequisite learning. Teach the LLM the prerequisite knowledge by training knowledge
LoRA, parameterized by Atkaow, to minimize the next-token prediction loss on D
min L( +
A0 knnww) ):
A A0aa
where [(0; D) represents the cross-entropy loss of the model computed on dataset D.
Step 2: Supervised fine-tuning (SFT). Equip the LLM with the downstream task skill by training a skill LoRA, parameterized by A0.m. to minimize the task loss Dy, with the knowledge LoRA present and frozen:
min (r A0know + A0 Dr); 26am
(2)
where (0 represents the downstream task loss of the model 0 computed on dataset D. In biography generation, Ly is also the cross-entropy loss on the ground-truth biographies.
Here an intuitive explanation of how this works. In the SFT stage, the presence of the knowledge LoRA ensures that the skill LoRA is always able to ground its generation on the internal knowledge of LLMs, so it would not have the incentive to fabricate ungrounded facts. Also, since the knowledge
LoRA has already absorbed all the knowledge information in Dy, the skill LoRA does not need to re-learn such knowledge and thus can focus on learning the skill information in Dr-
As shown in Figure T (right), during inference, the knowledge LoRA is dropped, and only the skill LoRA is retained, i.e., the LLM weights become A0 We hypothesize that this can guide A0wn to ground its answer on the original pre-trained knowledge in A.. not the additional knowledge in A This is particularly necessary if the knowledge in A0www is fictitious (a case to be introduced interferes with the original pre-trained knowledge.
in Section 3.3
One caveat, however, is that there exists an obvious gap between training and inference - A present during training. but absent during inference. It is unclear whether A0wa can generalize its knowledge grounding from the knowledge in 2 to the pre-trained knowledge. Fortunately, ample experiment evidence shows that the generalization is successful, as discussed in Section 4.2.
3.3 VERSION PREREQ- TUNEE WITH FICTITIOUS SYNTHETIC DATA
Our preliminary experiments show that the disentanglement mechanism in PREREQ-TURE is so powerful that even if (Dinww Dr) only contain fictitious synthetic data, i.e., biographies of fictitious people, the skill LoRA can still learn to generate biographies of real people once the knowledge LoRA (which absorbs the fictitious knowledge) is dropped. Inspired by this, we propose an upgraded training strategy, called multi-version PREREQ-TUS -UUN,, to further enhance the grounding of an LLM's outputs on its internal knowledge.
Specifically, rather than having only dataset pair (Danow, Dr). we leverage fictitious synthetic data to create K dataset pairs, U (D D(+)). Different knowledge datasets D(A) contain different versions of knowledge of the same set of fictitious entities. Different task datasets D'tt) con- tain the same set of questions, but with different ground-truth answers matching the corresponding knowledge in the knowledge set. For example, consider fictitious person named Avery Linwood. Assume that we create two versions of knowledge about Avery Linwood in two knowledge datasets. ((I) know only contains the birth year of Avery Linwood; (22)) only contains the birthplace of Avery Linwood. Then, the corresponding two task datasets, D. and DD(2) both ask the same question 'Generate a biography for Avery Linwood, ", but ground-truth answer in the former only talks about the birth year whereas the latter about the birthplace.
With the multiple versions of datasets, the training steps of PREREQ- TUNE are modified as follows.
Prerequisite learning. A different knowledge LoRA is trained on each of the different knowledge datasets:
Step 1:
min L(O, (C
(Dl)) (+) know
) Wk E (L,-..,K}
Step 2: SFT. Only one skill LoRA is trained, but each time with a different knowledge LoRA present. When knowledge LoRA (k))) is present, the skill LoRA is trained produce ground-
4

---

Published as a conference paper at ICLR 2025
Step I: Generate seed
Step 2: Get individual statements
Step Sample different knowledge datasets
Step 4: Construct task dataset
biographies
Dj 0 Biography with $q, Sg
Fictitious Biography
D know (5))8s)
Real Biography
GPT
Breakdown
(2) D know (52)
Dj 00 Biography with S2
Fictitious Hiogmphyy
Statements:
know (8a, S5.
2 Biography with S3, S5,
Random Draw
Prompt GPT
Figure 2: Procedure creating multi-version dataset pairs for the biography generation task.
truth answers in D which match the knowledge stored in A0
min AP
Cr
(A(() know
+ saii D
Since the K versions of the downstream task datasets contain the same questions but with different answers, the skill LoRA forced to link the different answers to the different knowledge LoRA versions. Thus this method can force groundedness to LLM's internal knowledge.
Multi-version POFREO-TUNE can only be enabled by fictitious synthetic data, not real data. This because fictitious knowledge, once removed from the knowledge LoRA, is guaranteed to be un- known to the LLM; whereas real knowledge, even if removed from the knowledge LoRA, may still exist in the pre-traing pre-trained knowledge. Thus using fictitious synthetic data ensures definitive control over what the LLM knows and does not know.
3.4 DATASET CONSTRUCTION
To generate a dataset pair (Diow, Dr), there are two strategies, top-down and bottom-up.
Top-down strategy. When the task dataset Dy is already available, for example when it is given real dataset, we can deduce the knowledge dataset Danow from Dr. In the biography generation task, for each biography in Dy, we break it into individual statements, e.g., 'John Estes was born in 1987', 'John Estes was born in California, USA' etc. This can be done by prompting an external instruction- tuned LLM (e.g., GPT-4), or using simple rule-based methods. Each statement then paraphrased M times to ensure that the knowledge LoRA learns the knowledge rather than memorizing the statements. All the paraphrased statements form the Droon.
Bottom-up strategy. To construct a fictitious dataset pair, we first construct Denow and then Dy. The fictitious knowledge can be generated using rule-based templates summarized from fictitious articles (details to follow). Then prompt the external instruction-tunel LLM to generate the task data for Dr. In the biography generation task, this involves asking the external LLM to generate biography of target entity based on all the statements concerning the target listed in Dkoow.
Generating multi-version dataset pairs. In the biography generation task, we combine top-down and bottom-up up strategies to generate multi-version (D(h) (())) pairs, as shown in Figure First, we prompt the external LLM to generate biographies of fictitious entities, using real Wikipedia biogra- phies as references. We ensure the fictitiousness of these biographies by filtering out those whose entity names coincide with Wikipedia entries These biographies are called seed biographies. Sec- ond, the seed biographies are broken into individual statements as in the top-down strategy. Third, the statements are sampled into different subsets to form different knowledge datasets (D(A)) For each fictitious entity, we first sample the number of facts (different paraphrases are considered the same fact) to include, and then sample the statement subset accordingly. Finally, the task datasets (Df)) are constructed from the corresponding knowledge datasets with the bottom-up strategy.
We also tried filtering by the number of occurrences of the name in the public pre-training corpus hut observed similar performance.
5

---

Published as a conference paper at ICLR 2025
Statement-basee v.s. passage-based knowledge. The knowledge dataset can take different forms. In the method above, it takes the form of individual statements. An alternative form would be short passages, which can be generated with just one additional step top of the statement-based knowl- edge. After the statement the statements of the same entity into one passage. Each passage is also paraphrased M times. A comparison between the two knowledge forms is presented in Appendix D.6
knowledge is created, we prompt the external LLM to summarize
3.5 EXTENSION TO ANSWER ABSTENTION
So far, we have described how to teach LLMs to properly respond when knows the answer. Op- tionally, PREREQ- TUNE can be extended to also teach LLMs to respond with 'I don't I know' when it is not familiar with the answer, by introducing unfamiliar examples into the dataset pair (Drow; Dr). Recall that each knowledge piece in Daow is paraphrased M times to ensure familiarity. Therefore, we can vary the number of paraphrases to create variations in familiarity. The knowledge pieces whose number of paraphrases drops below threshold (including zero) identified as unfamiliar knowledge, Then, we locate all the questions in D7 that depend on the unfamiliar knowledge and modify their answers to 'I don't know'. Note that unfamiliar knowledge can only be created from the fictitious knowledge, not the real knowledge, otherwise the LLM may have also learned the knowledge through pre-training. The similar approach can also be leveraged to train the model to express internal uncertainty, which will be elaborated in Section 4.6
3.6 EXTENSION TO OTHER TASKS
Although PREREQ-TUNE is described in the context of biography generation above, the method is readily generalizable to other tasks, with only slight variations in how the datasets are created (Section 3.4). For other long-form generation tasks that involve generating passages about a given entity, only minor change needed in the prompt for the external LLM used for generating the datasets (e.g., changing the word 'biography' to 'summary'). For short QA tasks, in the top-down strategy, rather than asking the external LLM to break the biographies into multiple facts, we ask to rewrite each QA pair in the task dataset into one statement, which forms the knowledge dataset. In the bottom-up strategy, fictitious knowledge and QA pairs are created by filling a pre-defined template containing an entity name field and some attribute fields, following the format in existing datasets (Mallen al. 2023). Further details can be found in Appendix B.
EXPERIMENTS
We evaluate PREREQ-TUSE -TUNE on two long-form generation tasks, biography generation and long- form medical QA, as well as two QA tasks, short factoid QA and multihop QA. The experiment settings and main results are presented in Sections 4.1 and 4.2 Subsequent sub-sections present more in-depth studies of PREREQ-TUNE
4.1 EXPERIMENT SETTINGS
Datasets. For long-form generation, we follow existing works to evaluate on biography generation and medical QA tasks (Tian et al. 2024). Both tasks require generating a long summary for the asked entity (a person for the former and a medical entity for the latter). For persons, we use the in- put instruction 'Generate biography for {person) and for medical entities, the instruction is 'Tell me about { {medical entity).. Since the data in Tian et al. (2024) is unavailable, we collect our own data and ensure no overlap of entities between training, validation, and test sets. For both tasks, we use the Wikipedia page as the ground-truth response in the training set. Additionally, for biography generation, we use the 183 labeled persons in Min et al. (2023) as test set keep consistent with prior works (Lin et al. 2024). For QA, we evaluate on PopQA (Mallen et al.) 2023) and HotpotQA (Yang et al. 20.8). PopQA contains factoid questions about 16 relations, e.g., 'Who was the director of Breaking Bad?" In preliminary experiments, tions in the dataset (e.g., the question 'In what country is Oxford?' is asking about a city in Ohio, USA), so clarify those questions and remove any that remains ambiguous. HotpotQA contains questions that require multiple reasoning steps to obtain the final answers. The questions require two types of reasoning bridging and comparison. Bridging questions involve asking facts about
notice substantial amount of ambiguous ques-

---

Published as a conference paper at ICLR 2025
Table 1: Performance for long-form generation tasks (persons and medical entities) and short QA.
method to show the impact of prerequisite learning. I; numbers different from Ghosal al. (2044 because we process ambiguous questions; see Appendix D.2 for results the original data. " lower than original paper because the original model only generates 2.7 claims.
:: trained with the same hyperparameters
Persons
Medical Entities
PopQA
HotpotQA
Comp. Acc.
Bridg. Acc. f
Acc.
# Claims
Acc. f
Claims
Acc.
SFT
32.70
20.8
69.94
9.2
46.42
61.75 59.51
25.99 24.44
POPULAR Ghossl et al.
2024
41.16
15.4
65.92
8.1
45.31
FLAME (Lin el al. 2024
30.32
18.2
67.92
9.8
FACTTUNE (Tian et al.
2024
31.93
19.6
69.13
RL Kang et al. 2024
33.20#
20,9
70.03
9.0
SFROPT
34.75
19.7
67.98
9.0
SFT SFTdecitioss+
15,44
20.6
64.44
8.9
44.98
51.96
10.27
PREREQ TUNE
45.30
16.0
74.35
9.1
47.91
65.02
26.22
bridging concept, e.g., 'Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?' Comparison questions involve comparing the factual information about two concepts, e.g., 'Who was born first, Pablo Trapero or Aleksander Ford?' For all questions, we generate the ground-truth chain-of-thought (COT) answers from the HotpotQA's labeled facts. For example, for the above bridging question, the reasoning chain is 'Guitars for Wounded Warriors was recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. New Paltz is located in Ulster County.' Please see Appendix for details of all datasets.
FActScore (Min et al. 2023 evaluate the generated summaries, which decomposes summaries into independent claims and verifies each claim against relevant Wikipedia paragraphs. We report the aceuracy the percentage of correct claims among all generated claims, averaged across all test examples. In initial evaluation, we observe that mod- els sometimes hack the metric by generating subjective or generic claims (e.g., 'She continues to influence and inspire new generations'). We thus filter the claims to only keep those that present objective and concrete information. For Q4, we follow prior works using accuracy measure hallucinations (Gekhman et al. 2024). where correctness is based on exact match with ground-trun
Metrics. For long-form generation,
Baselines. We consider five baselines. O SFT that performs the standard supervised fine-tuning on the given task dataset D7. @ POPULAR (Ghosal et al. 2024) that only performs SFT on subset of DD which the model has knowledge of. This subset is selected either by checking the base model's accuracy the questions or the monthly views of the entity's Wikipedia page. @ FLAME (Lin ct al. 2024). which prompts the base model 0, to generate summaries using in-context learning and uses these self-genne celf-generated summaries as ground-truthe for SFT. O FACTTUNE (Tian et al. 2024 that performs DPO (Rafailov et al.) 2023) on top of O, where the preference pairs are collected from the base model's 's sampled outputs (generated by in-context learning in ① and annotated with FActScore. @ RL (Kang et al. 2024). which runs PPO (Schulman et al. 2017 on top of and the reward model is trained on FActScore annotations of the self-generated summaries in ?
Implementation details. We use Llama-3-8B (Llama Team 2024) as the base LLM and LoRA fine-tune all baselines and our method. Additional results using larger models such Qwen2.5 14B (Qwen Team 2025) and Gemma2 27B (Gemma Team 2024) are presented in Appendix [D4] We search training steps, learning rate, and LoRA rank on the validation set for all methods. For long-form generation, we use multi-version PREREQ-TUNE on completely fictitious data, whereas for short QA, we use the basic version on a mix of fictitious and real data. For fair comparison, all methods access the same real task dataset Dy. For baselines @- we further ensure the total number of training examples matches our method (i.e., our fictitious data contains the same number of entities Dr, with the same number of responses per entity).
4.2
MAIN RESULTS
Table presents the main results, where PREREQ-TUS TUNE achieves the best performance across all four datasets, especially in long-form generation tasks. It outperforms baselines like FACTTUNE
7

---

Published as a conference paper at ICLR 2025
and RL, which use the evaluation metric during training. This suggests the benefits of PREREQ- TUNE, where explicitly training for alignment between the model's outputs and internal knowledge provides more direct signals for learning groundedness. The results on HotpotQA also suggest that PREREQ-TUNE can be generalized to more complex tasks that involve reasoning.
Notably, recall that for long-form generation, PREREQ- TUNE was trained completely fictitious data, and yet it still outperforms all baselines trained on real data. This shows the strong disentan- glement power of PREREO-TUNE. To further verify this, we introduce variant approach, called SFT which is also trained on the same fictitious data, but with the prerequisite learning stage removed. The corresponding results in Table show the worst performance, which indicates that is the prerequisite learning that turns the harmful fictitious data into a panacea.
One caveat is that the fictitious data was created using GPT-4, so it is possible that PREREO-TUNES perfor- mance advantage is due to distilling the stronger LLM. To rule out this possibility, we introduce another baseline, SFTOPT, which is fine-tuned on GPT-4 generated para- phrases of the ground-truth responses in the given dataset Dr. The results in Table T show a significant advantage of SHEREQ-TUNE over SFTGPT which confirms that it is not the distillation of GPT-4, but the disentanglement and groundedness designs that contribute the superior performance of PREREO-TUNE. Additionally, we also evaluate PREREQ-TUNE when GPT-4 replaced with Llama when constructing the synthetic data. Results in Appendix D.I show similar performance. demonstrating that our method is applicable using only open-source LLMs.
50
Ours
Popular
Accuracy
SFTOP
40
35
10.0
12.5
15.0 17.5 Claims
20.0
Figure 3: Accuracy on biography gen- eration under different numbers of gen- erated claims.
Our initial explorations reveal negative correlation between accuracy and the number of generated claims. To control this interference, we perform another experiment on the biography generation task where we vary the length of ground-truth responses in the training set to get models that generate different numbers of claims. Results in Figure show that our method consistently achieves the best performance across various numbers of generated claims.
Finally, Appendix D.3 shows that our models trained on one domain (e.g., biography generation) can generalize to another domain (e.g., long-form medical QA), which suggests the generalizability of the knowledge grounding ability. Appendix D.5) compares SFT and PREREQ- TUNE when trained on familiar and unfamiliar data respectively, verifying that our method outperforms SFT by larger margins when trained on unfamiliar data. 14 shows some sample outputs.
4.3 THE KNOWLEDGE GROUNDING OF PREREQ-TUNE
Although Section 4.2 shows the superior performance of PREREQ TUNE in hallucination reduction, it is still unclear whether PREREQ- truly learns ground the response the knowledge in knowledge LoRA as designed. To verify this, we design a knowledge grounding test on the QA task, where we fix the skill LoRA as the one trained for the QA task in the main results, but create two test knowledge LoRAs, ( know and Ae know The two test knowledge LoRAs learn two conflicting versions of knowledge about the same fictitious entities. For example, both knowledge LoRAs learn about the fictitious person Mira Telka, but version says they are an astrophysicist, and version 2 says they are an architect. It is worth emphasizing that the skill LoRA has never seen these test entities, nor the two test knowledge LoRAs, during training.
We then ask the LLM questions about these test fictitious entities with or without the test knowl- edge LoRAs Table [ shows the accuracy of L.MM's response evaluated against the two conflicting versions of ground-truth answers (Acc. V1 and Acc. V2). As can be observed, when one of the two knowledge LoRAS is plugged in (first two rows), LLM is able to generate the correct answer matching the corresponding knowledge LoRA over 90% of the time. When the knowledge LoRA is removed (third row), the LLM is unable to answer either version, whose accuracy is as low as the SFT baseline (last row) which has never seen any fictitious data. These results are clear evidence that the L.LM answers questions faithfully based the knowledge provided by the knowledge LoRA.
8

---

Published as a conference paper at ICLR 2025
Table 2: QA accuracy fictitious synthetic test data, which contains unseen questions for the skill LoRA A0am Accuracy is computed for two dif- ferent answers to the same question (VI and V2).
Table 3: Performance fictitious synthetic train- ing data. Memorized Entities measures the per- centage of named entities in the fictitious persons" biographics that are memorized.
Acc. V1
Acc. V2
QA
Bio Generation
0g + (g(1)
+ A0
94.83
6.90
Acc.
Memorized Entities
0o + 0
Anow A0
Snow
13.22
95,40
SFTfictiioos
58.01
32.63%
0a+ + Akm
15.52
5.17
0o + A0,m
3.99
10.79%
SFTeea
14.94
5.17
SFT 3.93
10.28%
We find these results very interesting, because they inform an innovative modular design LLM, where the knowledge LoRA serves as a plug-and-pla "knowledge flash drive" and the skill LoRA retrieves the knowledge to form an answer. It opens up the possibility of a novel retrieval aug- mented generation (RAG) paradigm, where the retrieval source is not the external documents, but the knowledge LoRAs, which may address the inference cost and context length challenges.
4.4 KNOWLEEDGE POLLUTION
By design, the skill LoRA does not need to learn any knowledge information because the knowledge LoRA already covers all. However, since the skill LoRA exposed to the knowledge information in the task dataset D during training, one might wonder whether any knowledge could accidentally get picked up by the skill LoRA, a phenomenon we refer to as knowledge pollution. Knowledge pollution, if present, would indicate that the disentanglement of PREREQ-TUS TUNE is incomplete, and undermine the reliability of the skill LoRA, especially when the new knowledge fictitious.
To test whether knowledge pollution is present, we plug the skill LoRA from the main results into the LLM,, remove the knowledge LoRA, and ask LLM questions about the training knowledge. Note that the main difference from Section 4.3 is that here the questions about fictitious training knowledge the skill LoRA has seen, whereas Section 4.3 is about unseen test knowledge. (ahle:]show the QA accuracy and the percentage of memorized entities for biography generation (the latter is computed as the percentage of named entities in the fictitious persons' knowledge that are generated by the model). The results indicate that without any knowledge LoRAs, the skill LoRA alone (second row) is unable to answer the questions correctly, and its behavior is similar to SFT which was trained on real data only and thus guaranteed no knowledge pollution. As a reference, SFT which trained directly the fictitious knowledge, achieves a much higher performance.
These results show no evidence of knowledge pollution, and thus verify the strong disentanglement power of PREREQ-TUNE. It also inspires potential privacy protection applications, where the skill LoRA can be trained on private data to learn the skills without memorizing any protected knowledge.
4.5 SCALING SYNTHETIC DATA
As discussed, synthetic data has the benefit of cheap scaling. We are thus interested in what would happen after such scaling. F shows the QA and biography generation performance (detailed settings in Appendix D.8) as the amount of fictitious synthetic data increases. The blue line shows the performance of PUERE which shows an increasing trend. The red line shows the per- formance of directly fine-tuning on the fictitious synthetic data without prerequisite learning, which exhibits clear degradation. This intriguing contrast confirms that while the fictitious synthetic data is inherently harmful, PREREQ- TUNE endows it with positive scaling effect.
4.6 ANSWER ABSTENTION AND VERBALIZED UNCERTAINTY
Finally, we evaluate the performance of the extended version of PREREQ-TUNE that also teaches the LLM to say 'I don't ion't know' (Section 3.5). Specifically, for the QA task, we create multi-version dataset pairs (k) ). such that for each question, one of the pairs turns the corresponding knowledge into unfamiliar knowledge (with number of paraphrases M 0), and the corresponding

---

Published as a conference paper at ICLR 2025
QA
Blography Generation
47.5
Accuracy
47
Accuracy
45.0
With Prereq-Tune Without Prereq-Tune
With Prereq-Tune Without Prereq-Tune
46
17.5
15.0
5000 10000 15000 20000
200
400
600
800
# Synthetic data
# Synthetic data
Figure 4: Performance as the number of synthetic data scales up.
answer into don't 't know' (IDK). We compare with baseline that splits the real data into unknown and unknown parts and trains the model for IDK responses on unknown data (Zhang et al. 2024). On the test set, for questions that the model knows, our method improves correctly answered questions from 53.20% 54.94% and reduces mistakenly abstained questions from 35.51% 33.64%, For unknown questions, our method increases the IDK responses from 85.32% to 85.63%.
The results above confirm that the number of knowledge paraphrases M can effectively control the familiarity of the knowledge, which inspires us use to enable more fine-grained expression for uncertainty. Specifi- cally, in addition to certain and unknown, we introduce third level of familiarity, unsure, in between by setting an- other threshold on M. The answers involving the unsure knowledge are rewritten Appendix for for more details). Figure 5 shows the dis- tribution of the three response types questions entities, across the log of monthly views of the entity's Wikipedia page, which is rough estimation of the tity's familiarity. As shown in the figure, the model's re- sponse type generally aligns with the entity's familiarity, indicating that the skill of verbalizing uncertainty, which learned only on fictitious synthetic data, can generalize reasonably to the base model's pre-trained knowledge.
0.3
Response Type
Cortain
Unsure
Density
Unknown
0.1
think it might be ...' (See
real
10
15
Log page views.
Figure 5: The distribution of each sponse type with respect to the log of monthly page views of the entities.
5 CONCLUSION
In this paper, we propose PREREQ-TUNE, a novel fine-tuning strategy to reduce LLM hallucina- tions. PREREQ- TUNE disentangles the learning of skills and knowledge by introducing a prerequi- site learning stage, which equips L.Mss with the necessary knowledge required for subsequent SFT. Moreover, PRERFO-TUNE can combined with fictitious synthetic data to improve the grounded- ness of LLMs generations. Experiments three datasets show that PREREQ-TUN -URRE outperforms strong baselines in reducing hallucinations, Further analyses also verify its disentanglement.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work of Yujian Liu and Shiyu Chang was partially supported by National Science Foundation (NSF) Grant IIS-233825,, NSF Grant IIS-2207052, and NSF Grant IIS-2302730. The computing resources used in this work were partially supported by the Accelerate Foundation Models Research program of Microsoft and Center for Al Safety Compute Cluster. Tommi Jaakkola acknowledges support from the MIT-IBM Watson AI Lab and the NSF Expeditions grant (award 1918839) Under- standing the World Through Code.
ETHICS STATEMENT
Our work aims enhance the reliability and trustworthiness of LLMs by reducing the hallucinations in their generations. However, while our method shows clear improvements over baselines, it does eliminate hallucinations. In fact, results in Table sample outputs in Table show that the fine-tuned model can still generate factually incorrect statements. Therefore, users should remain cautious when using our fine-tuned model and are strongly advised to verify its outputs through additional, trusted information sources. Furthermore, although our analyses in Section 4.4 show promising results for privacy protection, care should taken to ensure that no sensitive information inadvertently included in real-world deployments.
REPRODUCIBILITY STATEMENT
We have taken the necessary steps ensure the reproducibility of results. Specifically, Section discusses the general experiment settings in our paper. Appendix A provides the detailed steps collect and process the datasets used in downstream tasks. Appendix [ includes the detailed steps to construct the fictitious synthetic data used by our method. Finally, Appendix C and the supplementary material list the implementation details of our method and all baselines, including the codebase, training hyperparameters, evaluation details, etc.

---

Published as a conference paper at ICLR 2025
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024.
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, December 2023.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't know?, 2024.
I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. Factool: Factuality detection in generative ai tool aug- mented framework for multi-task and multi-domain scenarios, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and reduces hallucination in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lingnistics ACL 2024, August 2024.
Jason Weston.
Shangbin Feng, Weijia Shi. Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling LLMs' knowledge gaps with plug-in specialized language models. In The Twelfth International Conference on Learning Representations, 2024.
Leo Gao. Behavior cloning is miscalibrated, 2021.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2023.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?, 2024.
Google Gemma Team. Gemma 2: Improving open language models a practical size, 2024.
Gaurav Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction, 2024.
Yoav Goldberg. Reinforcement learning for language models, 2023.
Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Ar- jun Sawant, Chitta Baral, and Swaroop Mishra. TarGEN: Targeted data generation with large language models. First Conference on Language Modeling, 2024.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach them selves to program better. The Eleventh International Conference on Learning Representations, 2023.
Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang. A probabilistic framework for 1lm hallucination detection via belief tree propagation, 2024.
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination natural language generation. ACM Comput, Surv., 2023.
Erik Jones, Hamid Palangi, Clarisse Simões Ribeiro, Varun Chandrasekaran, Subhabrata Mukher- jee, Arindam Mitra, Ahmed Hassan Awadallah, and Ece Kamar. Teaching language models to hallucinate less with synthetic tasks. The Twelfth International Conference on Learning Rep- resentations, 2024.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate, 2024.
Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-skillmix: A powerful pipeline for IIm instruction tuning, 2024.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022.
Deren Lei, Yaxi Li, Mengya Hu, Mingyu Wang, Vincent Yun, Emily Ching, and Eslam Kamal. Chain of natural language inference for reducing large language model ungrounded hallucina- tions, 2023.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang. Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.
Junyi Li, Xiaoxue Cheng, Xin Zhao, Htan-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, December 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In shirth-seventh Conference on Neural Information Processing Systems, 2023b.
Sheng-Chich Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen tau Yih, and Xilun Chen. Flame: Factuality-aware alignment for large language models, 2024.
Meta Llama Team. The llama herd of models, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2023.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hal- lucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, December 2023.
13

---

Published as a conference paper at ICLR 2025
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese Susannah Young, Lucy Humpbell-Gilinghaam Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes, 2022.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, December 2023.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. In First Conference on Language Modeling, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted guestion-answering with human feedback, 2022.
Qwen Qwen Team. Qwen2.5 technical report, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, November 2021.
Shivehander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots, 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation--aummentedd language models. In The Eleventh International Conference on Learning Representattions, 2023b.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. SALMON: Self-alignment with instructable reward models. In The Twelfth International Conference on Learning Representations, 2024.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Represen- tations, 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers. Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the ólst Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), Toronto, Canada, 2023.
Sean Welleck. Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2020.
14

---

Published as a conference paper at ICLR 2025
Lilian Weng. Extrinsic hallucinations in llms. lillanwengggtthubbi., Jul 2024.
Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Tuliya Beloshapka, Wei-Hung Weng. Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating IIm hallucinations via conformal abstention, 2024.
Yuging Yang, Ethan Chern, Xipeng Qiu, Graham Neubig. and Pengfei Liu, Alignment for honesty, 2023.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods Natural Language Processing, 2018.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'I don't know'. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), June 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efratt, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023.
15

---

Published as a conference paper at ICLR 2025
Table Number examples in the real downstream task dataset Dy.
Persons
Medical Entities
PopQA
HotpotQA
Training
397
449
10,613
15,613
Validation
60
80
789
2,000
Test
183
200
2,152
5,405
A TASK DATASETS
This section presents the details of collecting and processing the real task dataset Dy for four tasks our experiments. Table shows the statistics for four datasets.
A.1 LONG-FORM GENERATION
For long-form generation tasks, collect real-world entities from Wikidata and use the first section of their Wikipedia page as the ground ground-truth response. Specifically, we collect entities within the cat- egory of "Human" for biography generation, and entities "class disease" and "medicinal prod- uet" for medical QA. Following prior works (Min et al. (202)), we consider entities whose Wikipedia page's monthly views are greater than 1,000 as popular. For biography generation, we format the mnstruction-responss pair as "Question: Generate biography for {entity). Answer: (response)'. For medical QA, the format is 'Question: Tell me about {entity). Answer: {response}'.
A.2 SHORT QA
For the short QA task, we evaluate on PopQA (Mallen et al. 2023) and HotpotQA (Yang et al, 2018). We randomly split PopQA data into training, validation, and test to ensure overlapping subjects, and we use the original split for HotpotQA. During the preliminary study, we find PopQA contains many ambiguous questions due to the ambiguity in the asked subject (e.g., the question 'Who was the director of Legacy?' could refer to multiple films with the same name Legucy). We thus clarify these questions by replacing the subject in the question with its Wikipedia title, which links to a unique real-world entity. The results on the original dataset are also reported in Section D.2 We format instruction-responss pairs 'Question: (question). Answer: {answer)' for training.
B DATASET PAIRS FOR PREREO-TUNE
We now describe the detailed procedure for constructing the dataset pair (Droww Dr) for PREREQ- TUNE on each task.
the statistics of the constructed datasets.
B. BASIC PREREQ- TUNE FOR SHORT QA
For the short QA task, we use a combination of fictitious and real data. For real data, we construct Danow from existing Dy using the top-down strategy. Specifically, for PopQA, since questions are created from knowledge triplets (subject, relation, object) using templates, we prompt GPT-4 con- vert each QA template into statement. For example, the QA pair 'Question: Who was the director of {subject) ? Answer: {object)' can be converted to a statement with '[subjecr} was directed by (object)." This template is further paraphrased 15 times by GPT-4, and the statement-based knowl- edge dataset is constructed by replacing the subject-object pairs in Dr into these templates. For HotpotQA, we prompt GPT-4 decompose the chain-of-thouge into individual statements and paraphrase each statement 15 times to construct the statement-bar knowledge dataset. For the passage-based knowledge dataset, we directly the Wikipedia page of the entities.
For fictitious data, we create Dr first and then convert it into knowledge dataset Disow using the following three steps. O We prompt GPT-4 to generate synthetic data that involve fictitious entities. Figures 8 and 9 list the detailed prompt for PopQA and HotpotQA respectively. @ We remove any generated data if any of the entities involved coincide with a real-world Wikipedia entry. This
16

---

Published as a conference paper at ICLR 2025
Table 5: Statistics of the synthetic datasets for PREREQ-TUNE.
Persons
Medical Entities
PopQA
HotpotQA
# entities
397
449
20,000
14,322 1
# knowledge versions
5
ensures the fictitiousness of the data. The resulting QA pairs form our synthetic downstream task dataset Dy. @ We construct the statement-base knowledge dataset following the same procedure as real data. For passage-based knowledge dataset, we prompt GPT-4 to generate a summary for each fictitious entity based on the answer in Dr.
B.2 MULTI-VERSION PREREQ-TUSE FOR LONG-FORM GENERATION
For long-form generation tasks about persons and medical entities, we use completely fictitious data for multi-version PREREQ-TUSE TUNE. The dataset construction combines both top-down and bottom-up strategies. In the following, we describe the detailed steps for biography generation. Datasets for medical entities created similarly using the prompts, except that mentions of persons changed to medical entities.
Step 1: Construct seed fictitious biographies. We prompt GPT-4 generate biography for fictitious person, using real persons' Wikipedia pages as references. The detailed prompt is listed in Figure TO We further filter out fictitious persons whose names coincide with a real Wikipedia entry. The remaining biographies are called seed biographies.
Step Get individual statements. We decompose the seed biographies into individual statements. We experiment with two decomposition methods. First, we prompt GPT-4 to break down the bi- ographies into atomic claims (instruction in Figure [3). Second, we break down the biographies into sentences and consider each sentence as an individual statement. The experiments in the main results adopt the first method, and Appendix D.6 further compares these two methods.
Step 3: Construet verti-version knowledge dataset. To create multiple versions of the knowledge dataset, we randomly sample the statements of person into different subsets and create a version of knowledge for each subset. Particularly, we employ two-step sampling procedure. First, we uni- formly sample the number of statements to cover, denoted as n. Then we sample n statements from the total set of statements without replacement. The resulting statements knowledge for the person. To create passage-
each paraphrased 15
times to construct one version of the statement- based knowledge, prompt GPT-4 generate summary for the person based on the subset of statements. The detailed prompt is listed in Figure Each summary is then paraphrased 5 times construct version of the passage-based knowledge for the person.
Step 4: Construct multi-version task dataset. The task dataset is created from the individual statements using the bottom-up strategy. Specifically, for each subset of statements, we use the prompt in Figure [] to instruct GPT-4 to generate a biography, which is considered as the ground- truth response for the corresponding version of knowledge.
B.3 MULTI-VERSION PREREQ-TURE TUNE FOR VERBALIZED UNCERTAINTY
We use completely fictitious data for the verbalized uncertainty experiment. We modify the fictitious data Section to create the multi version dataset pairs. Specifically, for each knowledge piece in Danow, create three versions of statement-based knowledge with different numbers of paraphrases. In ((1) each statement is paraphrased 15 times. This corresponds to the most familiar version of the knowledge. In 222) and ((3) each statement is paraphrased 6 and times respectively, which correspond to the moderate and least familiar versions. We then modify the task dataset so that the ground- response aligns with the model's familiarity. Particularly, in DI), we use the answer template I I'm sure the answer is ...". In DI? and DG3), we use the templates 'I think it might be and 'I don't i know, my guess is
17

---

Published as a conference paper at ICLR 2025
Table 6: Training hyperparameters in our experiments.
Persons
Medical Entities
PopQA
HotpotQA
Epochs
5, 10, 20, 30,
5. 10, 20, 20.30............ 30.........
80
3,4,5
3,4,5
learning rate
3c
5, Sc 5
3e 5c
3e
5, 5c 5
Ic
5, Jc
Batch size
128
128
512
512
LoRA
32, 64, 128
32, 64, 128
16, 12, 64
16,32, 64 2*r
LoRA
+
2
2+r
C IMPLEMENTATION DETAILS
We base our implementations on hlignmenthhandboor All experiments are conducted on 8 80GB NVIDIA A100 GPUs. During inference, number of training epochs, learning rate, and LoRA rank on the validation set for all methods. Table lists the hyperparameters we search.
greedy decoding for all methods. We tune the
On long-form generation tasks, we observe that models sometimes generate low-quality responses with many repetitions. We thus filter out checkpoints when more than 10% of their generations have a sep-rep-4 score higher than 0.2, as calculated in Welleck et al. (2020).
To evaluate generated summaries on long-form generation tasks, we use FActScore (Min et al. 2023) with GPT-4o-mini as the underlying LLM. To prevent models from exploiting the metric by generating generic or subjective statements, we modify the evaluation pipeline and add filtering step after decomposing the generated summary, so that only statements presenting objective and concrete information are evaluated. Figure 12 lists the prompt we use for this filtering.
C.I BASELINES
For POPULAR (Ghosa al. 2024), on long-form generation tasks, we train on the subset of entities whose monthly Wikipedia page views are greater than 1,000. On the short QA task, we follow definition in Gekhman al. (202)) split the training set into known and unknown questions and only fine-tune on known questions.
For FLAME (Lin al. 2024). follow settings in the original paper to sample outputs from the base model 0g using in-context learning with 5 demonstrations. We randomly sample training sampled for each entity match the
examples in D amount of data used by our method. To ensure the high quality training examples, we filter out samples with sep-rep-4 score higher than 0.2.
as demonstrations, 5 responses
For FACTTUNE (Tian et al. 2024), we evaluate the responses above using FActScore and collect (3) preference pairs for each entity, where the response with a higher score is considered as the chosen response and the other one as the rejected response. We set 0.1, learning rate as le 6, and train for 500 steps following (Lin al. 2024).
For RL (Kang et al. 2024), we use the official implementation and train the reward model using the above FActScore annotations.
C.2 BASIC PREREO-TUNE FOR SHORT QA
On short QA, we use combination of fictitious and real data. During prerequisite learning, we train separate knowledge LoRAs for fictitious and real entities. For both fictitious and real entities, we further train separate knowledge LoRAs statement-based and passage-based knowledge.
During SFT, we randomly sample a knowledge LoRA at each iteration, either statement- based or passage-based, and train the skill LoRA on top of it. Additionally, to mitigate the training and inference gap, we add regularization term that trains the skill LoRA on top of the base model,
without knowledge LoRA, which Icads to the following optimization problem:
min L (0o + A6 + Adamn Dr) + aLr(6o + Ad D 20am
(5)

---

Published as a conference paper at ICLR 2025
Table Generalization performance for long-form generation tasks.
Persons
Medical Entities
Acc.
# Claims
Acc. 1
# Claims
SFT
32.70
20.8
69.94
9.2
POPULAR (Gosal et al. 2024
41.16
15.4
65,92
8.1
PREREQ- -UNE
45.30
16.0
74.35
9.1
PREREQ- TUNE (transfer)
44.08
14.9
74.83
8.0
Table 8:
Accuracy on the original PopQA, without data cleaning.
Table 9: Performance of different formats for the knowledge dataset Dknow
Accuracy
QA Accuracy
Bio Generation Accuracy
SFT
36.90
POPULAR
36.85
Both
47.91
45.30
SFT
36.05
Statement- 47.58
38.75
PREREQ- TUNE
37.50
Passage-based
47.07
39.75
where Dipal is the subset in Dy that contains only real entities, and a is hyperparameter.
C.3 MULTI-VERSION PREREQ-TUEE TUNE FOR LONG-FORM GENERATION
On long- form generation tasks, we use completely fictitious data. For each knowledge dataset ((k) train two knowledge LoRAs, for statement-based and one for passage-hased.
know>
During SFT, at each iteration, we first uniformly sample a version from all possible versions, k ~ u((1...., K)). Then we randomly select either the statement-based passage-base based knowledge LoRA for this version, and train the skill LoRA on top of it using the task dataset D(L))
D ADDITIONAL RESULTS
D.1 PERFORMANCE OF PREREQ-TUNE USING LLAMA
We further evaluate PREREO-TUME TUNE when Llama-
3.1-70B-Instrun (Llama Team 2024 is used to con- struct the fictitious synthetic data instead of GPT-4,
50
Results in Figure [ show that using Llama achieves similar performance with GPT-4, demonstrating that PREREQ TUNE can be applied when only open- source LLMs are available.
Accuracy
45
40
GPT-4
LLaama
Popular
35
SFT
D.2 PERFORMANCE ON ORIGINAL POPQA
10.0
12.5
15.0
17.5
20.0
Table shows the results on the original PopQA dataset, without clarifying questions. Although the numbers are different from those in Table the gen- eral trend similar: TUNREQ-TUNE achieves the best performance whereas fFffictiioos is the worse, proposed prerequisite learning stage.
# Claims
Figure Accuracy on biography generation under different numbers of generated claims.
which again demonstrates the impact of the
D.3 GENERALIZABILITY OF PREREQ-TUNE MODELS
To study whether the knowledge grounding enhanced by approach generalize across dif- ferent domains, we conduct an additional experiment where we perform PREREQ-TUNE on one domain and test another. Specifically, we evaluate the models trained for biography generation
19

---

Published as a conference paper at ICLR 2025
Table 10: Performance on biography generation and multihop QA tasks using Qwen2.5 14B and
Gemma2 27B as the base LLMs.
Bio Generation
HotpotQA
HotpotQA
Comp. Acc. f
Bridg. Acc.
Comp. Acc.
Bridg.
Acc. f
Claims
Acc. f
Qwen2.5 14B
Gemma2 27B
SFT
25.93
19.3
66.70
23.77
68.75
32.06
POPULAR
33.98
13.8
65.02
22.85
66.51
30.53
PREREQ TUNE
39.24
14.9
67.44
24.83
68.94
32.86
Table 12: Results on biography generation when training and testing on data of different familiarity.
Test all
Test unfamiliar
Train all # Claims
Train on fictitious
Train all
Train fictitious Acc.
Acc.
Ace.
# Claims
Acc. 1
# Claims
# Claims
SFT
32.70
20.8
15.44
20.6 16.0
16.80
17.5
8.74
17.7
PREREQ-TUNE
45.59
16.0
45.30
25.16
13.4
22.97
13.6
Difference +12.89
+29.86
+8.36
+14.23
and long-form medical QA tasks our main experiments (Table [ e-g., we evaluate our biography generation model medical QA and our medical QA model on biography generation. The results are shown in Table As can be observed, the model trained with our method can be transferred to different domains (row PREREQ- TUNE (transfer)), and matches the performance our method in distribution (row PREREQ- Tune), outperforming all other baselines in distribution. These results indicate that the knowledge grounding ability can be generalized across different domains.
D.4 GENERALIZATION TO LARGER MODELS
In addition to Llama3 8B used in the main experiments, also evaluate method larger models including Qwen2.5 14B and Gemma2 27B. Toslo the results on bi- ography generation and multhhop QA tasks. As can observed, our method achieves the best performance, which indicates its generalizabil- ity across different base LLMs. Moreover, we repeat similar experiments as in Table to ver- ify if our method disentangles the learning of knowledge and skills. Results on Qwen2.5 14B are shown Table I1 which demonstrate that without the knowledge LoRA, the skill LoRA alone cannot answer questions correctly, and its performance is similar to model that has never the fictitious data (SF)
Table 11: Performance on fictitious synthetic training data using Qwen2.5 14B as base LLM.
HotpotQA Acc.
Bio Generation
Memorized Entities
SFT fistitoio
52.50
31.21%
O. + A0
11.98
12.41%
SFTea
10.33
10.56%
D.5 IMPACT OF TRAINING DATA FAMILIARITY
Since PREREQ- TUNE disentangles the learning of knowledge and skills, expect method to be more robust and outperform SFT by larger margins when trained on unfamiliar data. We now explicitly verify this claim by comparing models trained different data. Specifically, conduct experiments on multihop QA and biography generation tasks to investigate the influence of data familiarity in training and testing sets respectively.
For both datasets, we consider two different training settings: on the original training set (and fictitious data for our method). 2 Train fictitious data: We train
Train on all data: We train models
20

---

Published as a conference paper at ICLR 2025
Table 13: Results on HotpotQA when training and testing on data of different familiarity.
Test on all
Test unfamiliar
Train on all
Train on lictitions Brid. Ace.
Train on all Brid. Acc.
Train on fictitions
Comp. Acc.
Brid. Acc.
Comp. Acc.
Comp. Acc.
Comp. Acc.
Brid. Acc.
SFT
61.75
25.99
51.96
10.27
58,94
21.20
46.63
7.67
PREREQ-TUNE
65.02
26.22
58.02
20.93
63.05
21,09
54.25
17.28
Difference +3.27
+0.23
+6.06
+10.66
+4.11
-0.11
+7.62
+9.61
models only on our fictitious synthetic datasets. Because the entities in these datasets are fictitious, we ensure that models are unfamiliar with these datasets.
For each trained model, we evaluate it real-world questions from two testing sets: Test on all data: We test models on the original testing set. Test on unfamiliar data: We test models on a subset of data that they unfamiliar with. Specifically, we identify the QA pairs where at least one of the entities in the reasoning chain has a monthly Wikipedia page view of less than 500, and we consider these questions unfamiliar (for biography generation, we simply identify persons with small monthly page views).
Tables 12 and 13 show the results for biography generation and multihop QA tasks respectively. There are two observations. First, SFT performance drops significantly when training on fictitious data. This verifies our motivation that if fine-tuning data contains unfamiliar examples, perform- ing SFT would encourage hallucination. By contrast, our method has much smaller performance degradation, since it reduces the knowledge inconsistency. Second, both methods have worse per- formance when tested on unfamiliar data. In this case, the model has little knowledge about the entities, leading to limited performance even if it excels at knowledge grounding.
D.6 ABLATION STUDY
We now investigate the influence of three design choices in PREREQ-TUNE.
Data formats in the knowledge datasets. Table shows the performance of only using statement- based, passage-basee or both formats on short QA and biography generation tasks. As can be observed. combining both formats achieves better performance than using any of the formats alone. We hypothesize that knowledge is stored differently in knowledge LoRAs trained on these two for- mats, so training the skill LoRA to work with both formats improves its generalizability to different knowledge representations.
Methods decompose seed biographies. We compare two methods to decompose seed biogra- phies into individual statements. The first method prompts GPT-4 for decomposition (GPT-4-based), and the second method simply considers each sen- tence as statement (sentence-based). The main difference between the two is that GPT-4 breaks down biographies into atomic statements that cannot be further decomposed, whereas sentence-based de- composition may contain multiple facts within each statement (e.g., 'Emma Louise Carter (born May 22, 1965) is an American politician, entrepreneur, and philanthropist who served as the 48th vice presi- dent of the United States from 2009 to 2017 under President James Matthews.). Results in Figure 7 show that GPT-4-based decomposition outperforms sentence-based decomposition, which suggests that more fine-grained decomposition benefits the model in learning groundedness.
50
Accuracy
45
40
Prereq-Tune GPT-4 Prereq-Tune sentence
10.0
12.5 15.0
17.5
20.0
# Claims
Figure 7: Accuracy on biography generation with two decomposition methods.
Two-step training strategy. We evaluate another baseline to investigate the importance of the two- step training strategy in PREREQ- Specifically, we mix Dxnow and Dy and train a single LoRA on the mixed dataset. We use the exactly same datasets our method and tune the data mixture
21

---

Published as a conference paper at ICLR 2025
Input Instruction
Generate biography for Eliza Dushku.
Model Responses
Eliza Dushku (born December 30. 1980) is an American actress and model. She is known for her role as Kelly Reynolds the cable broadcast series Dollhouse (2009-2010). She also appeared the films True Lies (1994), The Tartuffe (2001), Bring It On (2000), Dog Days of Summer (2007). Tortured (2008), Gunshot Straight (2008). Miss Match (2003), Wrong Turn Tahoe (20019), Floise (2017), and Jules (2019), From its spin-oft Mayans M.C., Dushku also performed the voice of Athena in the video games True Lies (1994) and Jessica in the first two Guild Wars games.
SFT
to 2017, she portrayed Athena on FX's Sons Anarchy
Eliza Patrice Dushku (bom December 30, 1980) an American actress and model. She is recognized for her roles us FBI Special Agent Kimberly Price the TNT series Bull, Athena Syfy series Dollhouse,
the voice of Catwoman the animated film Batman: Assault on Arkham. Dushku has also appeared in films such as True Lies (1994). The Scorpion King (2002). Wrong Turn (2003), Jay and Silent Bob Strike Back (2001), Bring It On (2000), and Soul Man (2014), She recurring roles on the series Calling and Angel and has featured the video games True Crimes: Streets of L.A. and Dog Day Alternoon.
SPTOP
Eliza Patricia Dushku (born December 30, 1980) is an American for her roles as Faith in the action film series Buffy the Vampire Slayer (1992 2003) and its spin-off' television series Angel (1999-2004), well as her roles in True Lies (1994), Bring It On (2000), Jay and Silent Bob Strike Back (2001), Wrong Turn (2003), and Dollhouse (2009 (2009-2010). She also voiced Catwoman in the animated film Catwoman (2004) and voiced the character of Chelsea in the video game Mass Effect 3 (2012).
and model. She known
POPULAR
Eliza Patricia Dushku (born December 30, 1980) is an American actress, model, and activist. She is best known for her television roles as Faith on the action series Buffy the Vampire Slayer (1998-2003) and its spin-off series Angel (2000-2004), and as Jessica Angel True Lies (1994). She also starred in the Fox series Dollhouse (2009-2010) and appeared in the films True Lies (1994), The New Guy (2002). Bring It On (2000), Jay and Silent Bob Strike Back (2001), Soul Survivors (2001), Wrong Turn (2003), and The Alphahet Killer (2007).
PREREQ- TUNE
Example input instruction and model responses the biography generation task. We mark factually correct information in green and hallucinated information in red.
Table 14:
weights A and A2 to combine the two datasets (i.e., we select the best A from {0.1, 0.2, 0.3,.. ...)) On PopQA, the accuracy of this method is 40.66, which is worse than SFT (46.42) and our method (47.91). One possible explanation that the LoRA now needs to learn both skills and completely new knowledge, which leads to catastrophic forgetting of knowledge in the base LLM and worse performance when evaluated on real-world questions.
D.7 SAMPLE OUTPUTS
Table 14 shows model responses on an example for the biography generation task, which illustrates that our method generates less hallucinated content.
D.8 EXPERIMENT SETTINGS FOR SCALING SYNTHETIC DATA
Figure shows the performance when we scale up the amount of synthetic data. Specifically, for short QA, increase the number of synthetic questions. For biography generation, increase the number of fictitious synthetic persons and use sentence-based decomposition for PRERFO-TUNE.
22

---

Published as a conference paper at ICLR 2025
Given the following (subject, object) pairs that have the relation of relationh, generate 10 fictitious pairs that have the same relation. (example subject-object pairs)
Generate fictitious subjecta diverse possible and completely randem. Object be either fictitious real-world entities, but the relation should be valid.
Output each pair at line in JSON format with keys "subject" and "object
Directly generate results and nething else.
Figure 8: Prompt used for creating fictitious data for PopQA.
Given the following queation, ground-truth anawer, and related entities, generate a similar question but for fictitious entities.
Question: (question) Ground-truth answer: (answer)
Related entities: fentities)
Requizements:
The question should be about completely fictitious entities. The answer should start with short and concise reasoning chain and always end with "Final answer: ".
Senerate results in JSON format with keys "question", "answer", and illated_entities".
Figure 9: Prompt used for creating fictitious data for HotpotQA.
Here is the wikipedis biography for [real person) (wiki summary)
Using this as a reference, generate a biography for completely fictitious person.
Requirements:
The person should fictitious.
The biography should resemble provided Mikipedia biography
structure but pertain to fictitious person.
Do not mention "fictitious" or any other indication that the person is not real.
Generate outputs in valid jaon string format with keys "person" and biography".
Figure 10: Prompt used for creating fictitious biographies.
23

---

Published as a conference paper at ICLR 2025
Generate a biography for the person (fictitious person) based on the following facts.
Facts:
{set statements)
Requizements:
The biography must contain all information from the facts.
Only include information from the facts. Do not add any other information, You may rearzange and paraphrase the facts to make the biography more coherent.
Directly generate the biography and nothing else.
Figure 11: Prompt used for creating a biography from a set of statements.
Given the following claims about (entity), please select the cnes present concrete and specific information, Onit any claims that generic purely subjective.
Examples of clains with concrete information:
(posit iopsitive_ue aee_mmppp es)
Examples of generic or subjective claims: inegat.iv__exampeee
Claims:
[decomposed clains)
Requirements:
Copy each concrete claim after the and nothing else.
Figure 12: Prompt used for filtering out generic subjective claims in FActScore.
Please break down the following ng biography about (person) into indepandent facts. Focus on facts that present conerete and objective information. Biography:
(biography)
Requirements:
Generate atomic facts that cannet be further decomposed.
Include all information from the biography in the facts.
Facts should target (person), not other entities. Output each fact at separate 1ine after and nothing else.
Figure 13: Prompt used for decomposing seed biographies into individual statements.
24


Reviews:
[{"review_id": "fCb2IvVNsX", "reviewer": "Reviewer_D52y", "summary": "This paper introduces PREREQ-TUNE, a novel fine-tuning strategy to reduce hallucinations in large language models (LLMs). The key innovation is disentangling skill learning from knowledge acquisition by introducing a prerequisite learning stage before supervised fine-tuning. In this stage, a separate \"knowledge LoRA\" learns necessary background information, allowing the subsequent \"skill LoRA\" to focus purely on task-specific abilities without being misled by knowledge inconsistencies. PREREQ-TUNE can leverage fictitious synthetic data to further enhance output groundedness. Experiments on biography generation, medical QA, and short QA tasks show PREREQ-TUNE outperforms existing methods in reducing hallucinations. Additional analyses demonstrate its effectiveness in knowledge grounding, resistance to knowledge pollution, and ability to scale with synthetic data. The authors also explore extensions for answer abstention and expressing uncertainty. Overall, PREREQ-TUNE offers a promising approach to improve LLM factuality and opens possibilities for modular LLM designs.", "questions": "Could you show more detailed model architecture about how skill LoRA and knowledge LoRA interact with each other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Overall, I think the idea to learn a skill LORA that is capable of grounding is novel.\n2. The method is easy to follow with concrete running examples.", "weaknesses": "1. Although the overall method is interesting, the motivation is confusing to me. I think it is hard to argue that “if the tuning stage involves training examples that require knowledge that an LLM has not seen during pre-training, then the LLM would be misled to fabricate plausible but wrong answers to unfamiliar questions”. In a lot of previous literature, people use finetuning to adapt pretrained LMs to tasks [1][2], where I believe that LMs learn some knowledge during training. \n2. The citations in line 44 are not supportive. [3] mentions “A model’s knowledge and capabilities are learnt almost entirely during pretraining” in section 2 (The capabilities here look similar to the skills in the paper). [4] mentions that “pre-training is the main source of an LM’s capabilities”. In general, I don’t find it convincing that the tuning stage don’t involve knowledge learning.\n\n[1]. Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021, August). Taming pre-trained language models with n-gram representations for low-resource domain adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 3336-3349).\n[2]. Zheng, J., Hong, H., Wang, X., Su, J., Liang, Y., & Wu, S. (2024). Fine-tuning Large Language Models for Domain-specific Machine Translation. arXiv preprint arXiv:2402.15061.\n[3]. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., ... & Levy, O. (2024). Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oWQCka7lZ8", "reviewer": "Reviewer_G14i", "summary": "This paper proposed a new fine-tuning strategy called PREREQ-TUNE. The goal is to improve LLM's factuality through fine-tuning while minimizing the likelihood of introducing extra hallucination due to the knowledge inconsistency between pre-training and finetuning stages.\nThe basic idea is to disentangle models' learning of skills and knowledge by first introducing an additional prerequisite learning stage to focus on the knowledge that will be used in the subsequent SFT stage, so that the second SFT stage can focus only on learning the skills.\nExperiments show that PREREQ-TUNE outperforms existing baselines in improving LLM's factuality across short QA and long-form generation tasks.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper proposed a great idea to let an LLM enhance its grounding skills, meanwhile avoid introducing more hallucination due to the knowledge that the model may learn in the fine-tuning stage.\n\n2. The experiments very nicely show that the idea actually work on the test datasets, in particular, section 4.3 clearly shows that the skill LoRA can not answer the questions that require knowledge that the corresponding knowledge lora is not there. And I like the finding that the knowledge LoRA can serve as plug-and-play \"knowledge flash drive\".\n\n3. The method can even scale nicely due to the fact that it is tuned by synthetically generated data.", "weaknesses": "The method may be limited to simple facts. When the question is more complicated, for example requiring more than one fact or even reasoning to answer, the reasoning process itself (skill LORA)  may generate new facts that can introduce hallucination. In this case, the proposed method in this paper can be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LikkHoJIS", "reviewer": "Reviewer_ygrh", "summary": "The author believes that the issue of hallucination in language models largely arises from the discrepancies between the datasets used in the pre-training and fine-tuning stages. To mitigate this problem, this paper proposes a simple and effective fine-tuning method that first provides the necessary knowledge required for the subsequent stage, and then focuses on enhancing the model's \"skills.\"\nRich experiments focused on biography generation and doctor-patient question-answering tasks validate the effectiveness of this method in alleviating hallucinations.", "questions": "1. If enhancing the model with new \"skills\" requires re-executing data synthesis and knowledge enhancement, it may be excessively costly. Furthermore, when learning new \"skills,\" how should the LoRA parameters for previous \"skills\" be managed?\n2. The method mentioned in the paper appears to be quite effective, but based on the experimental results, it has not completely resolved the hallucination problem. What does the author consider to be the possible reasons for this? Additionally, what other factors might contribute to the occurrence of hallucinations in models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue of model hallucination is indeed a challenging problem that needs to be addressed.  \n2. Although the method is simple, it has proven to be highly effective from an experimental standpoint, with well-designed experiments supporting the arguments made in the paper and validating the necessity of each component.  \n3. The writing is fluid, and the use of figures and tables is appropriate, facilitating ease of reading.", "weaknesses": "1. The paper does not address the generalization ability of the models trained using this method.  \n2. When there are many \"skills\" to be learned, data synthesis may become quite costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0PEtXOe8gT", "reviewer": "Reviewer_Qkzs", "summary": "This paper proposes to use synthetic data to improve LLMs' factuality. Specifically, a \"knowledge\"-LoRA module is trained on some basic fictitious facts, and then a \"skill\"-LoRA module is trained to answer questions based on these facts. During inference, the knowledge-LoRA is dropped, and only the skill-LoRA is kept. Experiments show that the proposed approach can improve LLM's factuality across short-form QA and long-form generation tasks by a large margin.", "questions": "- How will the model perform if you simply fine-tune it with all fictitious data (D_know and D_T) without dropping the fictitious knowledge (in the basic version, not multi-version)? This seems to be a simple baseline that is good to see.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is simple and effective, outperforming SFT and other baselines for factuality-oriented fine-tuning.\n- Using different LoRA modules to better train the skill LoRA and flexibly remove the fictitious knowledge is very interesting and practical.\n- The ablations and analysis show that the skill-LoRA indeed learns to ground the response to the knowledge in knowledge-LoRA as desired.\n- The overall writing is of good quality and easy to follow.", "weaknesses": "- The novelty of the proposed approach is a bit limited. Previous methods also perform SFT in ways that are more aligned with the LLM's knowledge (e.g. the POPULAR method as cited), and the main novelty of the proposed method is to use synthetic data to ensure a higher level of groundedness. Given that the evaluated tasks are also simple ones involved with generating factual statements, it is not surprising that synthetic data could work well.\n\n- The generality of the method for more complex skills remains elusive. For example, how to correctly perform SFT on more challenging reasoning problems, where the LLM may not have enough reasoning skills to produce the responses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)