ICLR 2025

Published as a conference paper at ICLR 2025
FICTITIOUS SYNTHETIC DATA CAN IMPROVE LLM
FACTUALITY VIA PREREQUISITE LEARNING
Yujian Liu
UC Santa Barbara
yujianliu@ucsb.edu
Shiyu Chang
UC Santa Barbara
chang87@ucsb.edu
Tommi Jaakkola
MIT CSAIL
tommi@csail.mit.edu
Yang Zhang
MIT-IBM Watson AI Lab
yang.zhang2@ibm.com
ABSTRACT
Recent studies have identified one aggravating factor of LLM hallucinations as
the knowledge inconsistency between pre-training and fine-tuning, where unfa-
miliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs.
In this paper, we propose a novel fine-tuning strategy called PREREQ-TUNE to
address this knowledge inconsistency and reduce hallucinations. Fundamentally,
PREREQ-TUNE disentangles the learning of skills and knowledge, so the model
learns only the task skills without being impacted by the knowledge inconsistency.
To achieve this, PREREQ-TUNE introduces an additional prerequisite learning
stage to learn the necessary knowledge for SFT, allowing subsequent SFT to focus
only on task skills. PREREQ-TUNE can also be combined with fictitious synthetic
data to enhance the grounding of LLM outputs to their internal knowledge. Ex-
periments show that PREREQ-TUNE outperforms existing baselines in improving
LLM’s factuality across short QA and long-form generation tasks. It also opens
new possibilities for knowledge-controlled generation in LLMs. Our code is avail-
able at https://github.com/UCSB-NLP-Chang/Prereq_tune.git.
1
INTRODUCTION
Hallucination of large language models (LLMs) refers to the phenomenon where LLMs’ outputs
look plausible but diverge from real-world facts. It has become a major concern of LLMs, seriously
undermining their reliability and trustworthiness (Huang et al., 2023; Ji et al., 2023).
Recent research has unveiled one aggravating factor of LLM hallucination, which is the knowledge
inconsistency between the pre-training and tuning (e.g., instruction- or fine-tuning) stages (Gekhman
et al., 2024; Kang...