Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering
Authors: Shuzheng Si, Haozhe Zhao, Gang Chen, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Kaikai An, Kangyang Luo, Chen Qian, Fanchao Qi, Baobao Chang, Maosong Sun
Accepted at: acl25

Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering

Shuzheng Si*◊, Haozhe Zhao◊, Gang Chen◊, Cheng Gao♣, Yuzhuo Bai♣
Zhitong Wang♣, Kaikai An◊, Kangyang Luo♣, Chen Qian♣
Fanchao Qi*♣, Baobao Chang*, and Maosong Sun*♣♣♣
♣ Department of Computer Science and Technology, Tsinghua University
♣ Institute for AI, Tsinghua University ◊ Peking University ◇ DeepLang AI
* Jiangsu Collaborative Innovation Center for Language Ability

Abstract
Training LLMs on data containing unfamiliar knowledge during the instruction tuning stage can encourage hallucinations. To address this challenge, we introduce NOVA, a novel framework designed to identify high-quality data that aligns well with the LLM’s learned knowledge to reduce hallucinations. NOVA includes Internal Consistency Probing (ICP) and Semantic Equivalence Identification (SEI) to measure how familiar the LLM is with instruction data. Specifically, ICP evaluates the LLM’s understanding of the given instruction by calculating the tailored consistency among multiple self-generated responses. SEI further assesses the familiarity of the LLM with the target response by comparing it to the generated responses, using the proposed semantic clustering and well-designed voting strategy. Finally, to ensure the quality of selected samples, we introduce an expert-aligned reward model, considering characteristics beyond just familiarity. By considering data quality and avoiding unfamiliar data, we can utilize the selected data to effectively align LLMs to follow instructions and hallucinate less. Extensive experiments and analysis show that NOVA significantly reduces hallucinations and allows LLMs to maintain a strong ability to follow instructions.1

1 Introduction
Alignment is a critical procedure to ensure large language models (LLMs) follow user instructions (OpenAI, 2023a; Yang et al., 2024). Despite significant progress in LLM alignment and instruction tuning (Ouyang et al., 2022; Anthropic, 2022), state-of-the-art aligned LLMs still generate statements that appear credible but are actually incorrect, referred to as hallucinations (Ji et al., 2023; Huang et al., 2024). Such hallucinations can undermine the trustworthiness of LLMs in real-world applications (Si et al., 2023; Min et al., 2023; Rawte et al., 2023; Wei et al., 2024a).

Previous studies (Kang et al., 2024; Gekhman et al., 2024; Lin et al., 2024b) indicate that tuning LLMs on instruction data that contains new or unfamiliar knowledge can encourage models to be overconfident and promote hallucinations. In other words, once the knowledge in the instruction data has not been learned during the pre-training stage of LLMs, the fine-tuned LLMs tend to produce more errors when generating responses. Therefore, there is a dilemma in instruction tuning: On the one hand, the LLMs need to learn to follow user instructions during this stage, which is crucial for user interaction in real-world applications (Wang et al., 2023b; Chen et al., 2024b); On the other hand, using high-quality data (whether manually labeled or generated by other advanced LLMs) for instruction tuning can introduce unfamiliar knowledge to LLMs, thereby encouraging hallucinations (Kang et al., 2024; Lin et al., 2024b). Thus, a critical question arises: How can we align LLMs to follow instructions and hallucinate less during

![Instruction following ability on MT-Bench vs hallucination on LongFact. NOVA simultaneously aligns LLMs to follow instructions and hallucinate less.](page_1012_370_388_388.png)

Figure 1: Instruction following ability on MT-Bench vs hallucination on LongFact. NOVA simultaneously aligns LLMs to follow instructions and hallucinate less.

1 The data and code will be available at https://github.com/S1s-Z/NOVA. Email: ssz24@mails.tsinghua.edu.cn.
the instruction tuning stage?
Certain efforts (Lin et al., 2024b; Zhang et al., 2024b; Tian et al., 2024) apply reinforcement learning (RL) to teach LLMs to hallucinate less after the instruction tuning stage. For example, Zhang et al. (2024b) leverages the self-evaluation capability of an LLM and employs GPT-3.5-turbo (OpenAI, 2022) to create preference data, subsequently aligning the LLM with direct preference optimization (DPO) (Rafailov et al., 2023). However, Lin et al. (2024b) finds that such RL-based methods can weaken the model’s ability to follow instructions. These methods also necessitate additional preference data and API costs from the advanced LLMs, making them inefficient. Different from RL-based methods, an intuitive strategy to align LLMs to follow instructions and hallucinate less is to filter out the instruction data that contains unfamiliar knowledge for the instruction tuning. Unfortunately, previous studies (Liu et al., 2024a; Cao et al., 2024) solely focus on selecting high-quality data to improve the instruction-following abilities of LLMs. Even worse, these selected high-quality data may present more unknown knowledge to the LLM and further encourage hallucinations, as these data may contain responses with expert-level knowledge and often delve into advanced levels of detail.

Therefore, we introduce NOVA, which includes Internal Consistency Probing (ICP) and Semantic Equivalence Identification (SEI), a framework designed to identify high-quality instruction samples that align well with LLM’s knowledge, thereby aligning the LLM to follow instructions and hallucinate less. NOVA initially uses ICP and SEI to measure how well the LLM understands the knowledge in the given instruction and target response. For ICP, we prompt the LLM to generate multiple responses to demonstrate what it has learned about a specific instruction during pre-training. Then we use the internal states produced by the LLM to assess how consistent the generated responses are. If the internal states of these responses exhibit greater consistency for the instruction, it indicates that the LLM has internalized the relevant knowledge during pre-training. For SEI, we first integrate a well-trained model to classify the generated responses that convey the same thing into a semantic cluster. Next, we employ the designed voting strategy to identify which semantic cluster the target response fits in. This helps us find out how many generated responses are semantically equivalent to the target response, indicating how well the LLM understands the target response. If the target response matches well with the largest cluster, it shows the LLM is familiar with its content. Based on ICP and SEI, we can measure how well the model understands the knowledge in instruction data and avoid training it on unfamiliar data to reduce hallucinations. Lastly, to ensure the quality of selected samples, we introduce an expert-aligned quality reward model, considering characteristics beyond just familiarity, e.g., the complexity of instructions and the fluency of responses. By considering data quality and avoiding unfamiliar data, we can use the selected data to effectively align LLMs to follow instructions and hallucinate less.

We conduct extensive experiments to evaluate the effectiveness of NOVA from both instruction-following and hallucination perspectives. Experimental results demonstrate that NOVA significantly reduces hallucinations while maintaining a competitive ability to follow instructions.

2 Related Work

Hallucinations in LLMs. Hallucinations occur when the generated content from LLMs seems believable but does not match factual or contextual knowledge (Ji et al., 2023; Huang et al., 2024; Si et al., 2025). Recent studies (Lin et al., 2024b; Kang et al., 2024; Gekhman et al., 2024) attempt to analyze the causes of hallucinations in LLMs and find that tuning LLMs on data containing unseen knowledge can encourage models to be overconfident, leading to hallucinations. Therefore, recent studies (Lin et al., 2024b; Zhang et al., 2024b; Tian et al., 2024) attempt to apply RL-based methods to teach LLMs to hallucinate less after the instruction tuning stage. However, these methods are inefficient because they require additional corpus and API costs for advanced LLMs. Even worse, such RL-based methods can weaken the instruction-following ability of LLMs (Lin et al., 2024b). In this paper, instead of introducing the inefficient RL stage, we attempt to directly filter out the unfamiliar data during the instruction tuning stage, aligning LLMs to follow instructions and hallucinate less.

Data Filtering for Instruction Tuning. According to Zhou et al. (2023), data quality is more important than data quantity in instruction tuning. Therefore, many works attempt to select high-quality instruction samples to improve the LLMs’ instruction-following abilities. Chen et al. (2023); Liu et al. (2024a) utilize the feedback from well-
Figure 2: The process of NOVA. NOVA identifies and selects high-quality instruction data that aligns well with the LLM’s learned knowledge to reduce hallucination. Then it uses selected instruction data for training LLMs.

aligned close-source LLMs to select samples. Cao et al. (2024); Li et al. (2024a); Ge et al. (2024); Si et al. (2024); Xia et al. (2024); Zhang et al. (2024a) try to utilize the well-designed metrics (e.g., complexity) based on open-source LLMs to select the samples. However, these high-quality data always contain expert-level responses and may contain much unfamiliar knowledge to the LLM. Unlike focusing on data quality, we attempt to identify the samples that align well with LLM’s knowledge, thereby allowing the LLM to hallucinate less.

3 Methodology

In this section, we will detail our proposed framework NOVA as shown in Figure 2. Previous studies (Lin et al., 2024b; Kang et al., 2024; Gekhman et al., 2024) find that tuning LLMs on data containing new or unfamiliar knowledge can encourage models to be overconfident and further lead to hallucinations. Inspired by this finding, NOVA aims to filter out the unfamiliar instruction data for the instruction tuning, thereby aligning the LLM to follow instructions and hallucinate less.

3.1 Internal Consistency Probing

To comprehensively measure the LLM’s familiarity with instruction data, the first challenge is to evaluate how well the LLM understands the knowledge within the instructions. Prompting LLMs to generate multiple responses to the same instruction and measuring how consistent those responses are has been proven to be an effective way (Wang et al., 2023a; Chen et al., 2024a). This is because if LLMs understand the question and are confident in their answers, they will produce similar responses. A practical way to measure the consistency of free-form responses is to utilize lexical metrics (e.g., Rouge-L) (Lin et al., 2024c) or sentence-level confidence scores (e.g., perplexity) (Ren et al., 2023). However, these straightforward strategies neglect highly concentrated semantic information within the internal states of LLMs, and thus fail to capture the fine-grained differences between responses.

Hence, we propose Internal Consistency Probing (ICP) to measure the semantic consistency in the dense embedding space. For an instruction data \( s = (q, r) \), \( q \) denotes the instruction, and \( r \) denotes the target response. For instruction \( q \), we first sample \( K \) responses \( [r'_1, ..., r'_K] \) from a base LLM and apply few-shot demonstrations (Lin et al., 2024a) to ensure the coherence of generated responses. For \( K \) generated responses, we use the internal states of the last token of each response in the last layer as the final sentence embeddings \( E = [e_1, e_2, ..., e_K] \), as it effectively captures the sentence semantics (Azaria and Mitchell, 2023). We further utilize differential entropy (DE) to assess the semantic consistency in continuous embedding space, which is the extension of discrete Shannon entropy:

\[
\mathrm{DE}(X) = - \int_x f(x) \log(f(x)) dx.
\]

We process and treat sentence embeddings \( E \) as
a multivariate Gaussian Distribution \( E \sim N(\mu, \Sigma) \). Then, the differential entropy can be expressed as:

\[
\mathrm{DE}(E) = \frac{1}{2} \log((2\pi e)^d \det(\Sigma)),
\]

where \( \det(\Sigma) \) represents the determinant of the covariance matrix \( \Sigma \), \( d \) is the dimension of the sentence embedding, and \( e \) is the natural constant. \( \Sigma \) denotes the covariance matrix that captures the relationship between \( K \) different sentence embeddings, which takes the form:

\[
\Sigma = \frac{1}{K-1} \sum_{i=1}^K (e_i - \mu)(e_i - \mu)^T.
\]

Finally, we measure semantic consistency using \( \mathrm{DE}(E) \), term as \( F_{ins}(q) \) for a given instruction \( q \) in data \( s \). Also, \( \mathrm{DE}(E) \) in Eq.(2) simplifies to:

\[
F_{ins}(q) = \frac{1}{2} \log \det(\Sigma) + \frac{d}{2} (\log 2\pi + 1) = \frac{1}{2} \sum_{i=1}^d \lambda_i + G,
\]

where \( \lambda_i \) denotes the \( i \)-th eigenvalue of the covariance matrix \( \Sigma \), which can be easily calculated by singular value decomposition. \( G \) is a constant.

If the LLM is familiar with the given instruction, the sentence embeddings of generated responses will be highly correlated and the value of \( F_{ins}(q) \) will be close to \( G \). On the contrary, when the LLM is indecisive, the model will generate multiple responses with different meanings leading to a significant value of \( F_{ins}(q) \). In this way, we can exploit the dense semantic information to effectively measure the LLM’s familiarity with the instruction.

3.2 Semantic Equivalence Identification

Another challenge is to estimate the knowledge in the target response and measure the LLM’s familiarity with it, since the target response can contain expert-level and unfamiliar knowledge for the LLM. Training LLMs on such data can encourage hallucinations. Therefore, we propose **Semantic Equivalence Identification (SEI)** to measure the LLM’s familiarity with the target response by calculating how many generated responses are semantically equivalent to the target response. If the target response and more generated responses convey the same meaning, it indicates that the LLM is more familiar with it, thereby training the LLM on this target response will reduce hallucinations.

As the target response is manually labeled or derived from advanced LLMs (e.g., GPT-4) instead of generated by the LLM itself, the internal states of the LLM cannot effectively represent the target response. Thus, unlike utilizing internal states as the proposed ICP, we calculate LLM’s familiarity with target responses using the proposed semantic clustering strategy. In detail, we first cluster the generated responses that convey the same thing into a semantic cluster. This is because these responses are often free-form, and multiple generated responses can have the same meaning in different ways. Therefore, we employ an off-the-shelf natural language inference (NLI) model to cluster these responses. NLI models are trained to infer the logical entailment between an arbitrary pair of sentences. Thus, NLI models are well-suited to identify semantic equivalence, as two generated responses mean the same thing if you can entail (i.e. logically imply) each from the other (Kuhn et al., 2023; Jung et al., 2024). In this way, we can use an NLI model to consider two responses that can be entailed from each other as semantically equivalent responses. Specifically, we test each pair \( (r'_i, r'_j) \) of \( i \)-th and \( j \)-th generated responses as:

\[
F_{equivalent}(r'_i, r'_j) = \mathbb{I}\left\{ L_{NLI}(r'_i \Rightarrow r'_j) = L_{entailment} \land L_{NLI}(r'_j \Rightarrow r'_i) = L_{entailment} \right\},
\]

where \( L_{NLI} \) represents the predictions of the NLI model, \( L_{entailment} \) means the label of entailment relation. \( \mathbb{I} \) is the indicator function.

In this way, we can identify the semantic equivalence of each pair of generated responses and then cluster these generated responses \( [r'_1, ..., r'_K] \) into \( M \) different semantic clusters \( [c_1, ..., c_M] \), where \( m \)-th semantic cluster \( c_m \) contains \( k_m \) generated responses. Each semantic cluster \( c \) is a set of generated responses that convey the same thing. We further apply the NLI model to determine which semantic cluster the target response \( r \) fits in. Specifically, we use the model to test the target response \( r \) and each generated response \( r'_i \in [r'_1, ..., r'_K] \):

\[
F_{equivalent}(r, r'_i) = \mathbb{I}\left\{ L_{NLI}(r \Rightarrow r'_i) = L_{entailment} \land L_{NLI}(r'_i \Rightarrow r) = L_{entailment} \right\}.
\]

Using this method, we can determine how many generated responses in a semantic cluster are semantically equivalent to the target response \( r \). For semantic clusters \( [c_1, ..., c_M] \), the counts of such generated responses are \( [k'_1, k'_2, ..., k'_M] \). We use the votes in each semantic cluster to decide which
cluster the target response belongs to:

\[
\text{Index}(c_{target}) = \arg\max([\frac{k_1'}{k_1}, \frac{k_2'}{k_2}, ..., \frac{k_M'}{k_M}]).
\]

(7)

We calculate the ratio of the number of responses \( k_{target} \) in the target cluster \( c_{target} \) to the total number of generated responses as \( F_{res}(r) \):

\[
F_{res}(r) = \frac{k_{target}}{\sum_{m=1}^M k_m}.
\]

(8)

According to Eq.(8), when the LLM is familiar with the knowledge within the target response \( r \), most of the generated responses will have the same meaning as target response \( r \), thus the value of \( F_{res}(r) \) will be close to 1. On the contrary, if the target response contains unseen knowledge, i.e., none of the generated responses have the same meaning as it, the value of \( F_{res}(r) \) will be close to 0. To this end, we can effectively measure the LLM’s familiarity with the target response.

3.3 Ranking, Selecting, and Training

To comprehensively estimate the knowledge and consider both the LLM’s familiarity with the instruction and the target response, we calculate the ratio between \( F_{ins}(q) \) and \( F_{res}(r) \) for an instruction data \( (q, r) \) as the final score:

\[
F_{familiarity}(q, r) = \frac{F_{res}(r)}{F_{ins}(q)}.
\]

(9)

This score effectively measures how well the LLM understands the knowledge in instruction data. High \( F_{familiarity} \) values indicate that the knowledge in the data aligns well with the LLM, as they show that the generated responses are very consistent for a given instruction (i.e., low \( F_{ins}(q) \) values) and the generated responses are very semantically similar to the target response (i.e., high \( F_{res}(r) \) values). Based on the principle of filtering unfamiliar instruction data, the data with high \( F_{familiarity} \) should be selected to train the LLM.

However, our early experiments observed that selecting instruction data solely based on the LLM’s familiarity \( F_{familiarity} \) significantly reduces hallucinations but hinders the model’s ability to follow instructions. This is because considering only familiarity ignores other important characteristics of instruction data, e.g., the complexity of the instruction and the fluency of the response. Therefore, we further introduce an expert-aligned quality reward model to measure the data quality. We use an expert-labeled preference dataset (Liu et al., 2024b) which contains 3,751 instruction data to train a reward model (more details are shown in Appendix B). To take both familiarity \( F_{familiarity}(q, r) \) and quality \( F_{quality}(q, r) \) into consideration, we define the mixed rank \( R_{final}^{(i)} \) for \( i \)-th data as the average of the two ranks corresponding to the two metrics:

\[
R_{final}^{(i)} = \frac{1}{2}(R_{familiarity}^{(i)} + R_{quality}^{(i)}),
\]

(10)

where \( R_{familiarity}^{(i)} \) and \( R_{quality}^{(i)} \) refer to the ranks of the \( i \)-th data point in the degree of familiarity and quality. In this way, we can effectively consider data quality and avoid unfamiliar data.

Finally, we rank all the instruction data with their corresponding mixed rank \( R_{final} \) to select the top-ranked data, e.g., selecting the top 5% data to apply the supervised finetuning on the LLM. Based on the proposed NOVA, we can use the suitable data to effectively align LLMs to follow instructions and hallucinate less during the instruction tuning stage.

4 Experiment

In this section, we conduct experiments and provide analyses to justify the effectiveness of NOVA.

4.1 Setup

Instruction Dataset. We conduct instruction tuning with two different instruction datasets. Alpaca (Taori et al., 2023) contains 52,002 samples that are created by employing Text-Davinci-003 model (Ouyang et al., 2022) and Self-instruct framework (Wang et al., 2023c). Alpaca-GPT4 (Peng et al., 2023) further employs more powerful GPT-4 (OpenAI, 2023b) to get high-quality instruction data.

Evaluation. To evaluate our method comprehensively, we select widely adopted benchmarks for the targeted abilities. (1) Factuality hallucination benchmark: BioGEN (Min et al., 2023) and LongFact (Wei et al., 2024b); (2) Faithfulness hallucination benchmark: FollowRAG-Faithfulness (Dong et al., 2024), including 4 different QA datasets; (3) Instruction-following benchmark: MT-Bench (Zheng et al., 2023) and FollowRAG-Instruction. Comprehensive descriptions of tasks, datasets, and evaluation metrics are detailed in Appendix A.

Baselines. We compare several strong baselines, including (1) Vanilla Instruction Tuning: Vanilla
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">BioGEN†</th>
    <th colspan="3">LongFact‡</th>
    <th colspan="5">FollowRAG - Faithfulness§</th>
  </tr>
  <tr>
    <th>FactScore</th>
    <th>Respond</th>
    <th>Facts</th>
    <th>Objects</th>
    <th>Concepts</th>
    <th>Avg.</th>
    <th>NaturalQA</th>
    <th>TriviaQA</th>
    <th>HotpotQA</th>
    <th>WebQSP</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>42.4</td>
    <td>100.0</td>
    <td>17.1</td>
    <td>85.8</td>
    <td>80.3</td>
    <td>83.1</td>
    <td>40.5</td>
    <td>53.5</td>
    <td>16.0</td>
    <td>49.5</td>
    <td>39.9</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>47.2</td>
    <td>100.0</td>
    <td>15.6</td>
    <td>88.3</td>
    <td>81.2</td>
    <td>84.8</td>
    <td>43.5</td>
    <td>57.0</td>
    <td>17.5</td>
    <td>52.0</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>48.3</td>
    <td>100.0</td>
    <td>16.9</td>
    <td>87.8</td>
    <td>81.0</td>
    <td>84.4</td>
    <td>43.0</td>
    <td>58.0</td>
    <td>16.5</td>
    <td>52.5</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>48.1</td>
    <td>100.0</td>
    <td>21.0</td>
    <td>87.2</td>
    <td>80.5</td>
    <td>83.9</td>
    <td>41.5</td>
    <td>57.0</td>
    <td>15.5</td>
    <td>51.5</td>
    <td>41.4</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>47.9</td>
    <td>100.0</td>
    <td>16.2</td>
    <td>86.6</td>
    <td>79.1</td>
    <td>82.9</td>
    <td>42.5</td>
    <td>58.0</td>
    <td>16.5</td>
    <td>51.0</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>48.2</td>
    <td>100.0</td>
    <td>18.3</td>
    <td>88.6</td>
    <td>81.2</td>
    <td>84.9</td>
    <td>42.5</td>
    <td>56.0</td>
    <td>16.5</td>
    <td>51.0</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>NOVA - 5%</td>
    <td><b>50.3</b></td>
    <td>100.0</td>
    <td>17.9</td>
    <td><b>92.4</b></td>
    <td><b>82.7</b></td>
    <td><b>87.6</b></td>
    <td><b>46.5</b></td>
    <td><b>60.0</b></td>
    <td><b>19.0</b></td>
    <td><b>53.5</b></td>
    <td><b>44.8</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+2.9</td>
    <td>-</td>
    <td>+1.8</td>
    <td>+5.6</td>
    <td>+2.5</td>
    <td>+4.5</td>
    <td>+5.5</td>
    <td>+1.0</td>
    <td>+2.5</td>
    <td>+3.6</td>
    <td>+4.9</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>43.2</td>
    <td>100.0</td>
    <td>20.5</td>
    <td>86.5</td>
    <td>79.2</td>
    <td>82.8</td>
    <td>40.5</td>
    <td>60.0</td>
    <td>17.5</td>
    <td>53.5</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td>45.2</td>
    <td>100.0</td>
    <td>24.3</td>
    <td>87.1</td>
    <td>81.3</td>
    <td>84.2</td>
    <td>44.0</td>
    <td>59.5</td>
    <td>18.0</td>
    <td>48.5</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>45.8</td>
    <td>100.0</td>
    <td>27.1</td>
    <td>86.7</td>
    <td>80.4</td>
    <td>83.6</td>
    <td>43.0</td>
    <td>58.5</td>
    <td>17.0</td>
    <td>52.5</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>NOVA - 10%</td>
    <td><b>46.8</b></td>
    <td>100.0</td>
    <td>18.4</td>
    <td><b>89.1</b></td>
    <td><b>81.6</b></td>
    <td><b>85.4</b></td>
    <td><b>46.0</b></td>
    <td><b>63.0</b></td>
    <td><b>20.0</b></td>
    <td><b>59.0</b></td>
    <td><b>47.0</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+4.4</td>
    <td>-</td>
    <td>+3.3</td>
    <td>+2.6</td>
    <td>+1.3</td>
    <td>+2.3</td>
    <td>+5.5</td>
    <td>+4.0</td>
    <td>+2.5</td>
    <td>+5.5</td>
    <td>+4.7</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>42.2</td>
    <td>100.0</td>
    <td>19.4</td>
    <td>84.7</td>
    <td>80.7</td>
    <td>82.7</td>
    <td>43.5</td>
    <td>63.0</td>
    <td>23.0</td>
    <td>50.0</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>43.9</td>
    <td>100.0</td>
    <td>20.9</td>
    <td>86.4</td>
    <td>78.0</td>
    <td>82.2</td>
    <td>45.5</td>
    <td>61.5</td>
    <td>22.0</td>
    <td>48.0</td>
    <td>44.3</td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>44.3</td>
    <td>100.0</td>
    <td>23.4</td>
    <td>86.5</td>
    <td>80.1</td>
    <td>83.3</td>
    <td>45.0</td>
    <td>62.5</td>
    <td>21.0</td>
    <td>49.0</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>NOVA - 15%</td>
    <td><b>46.9</b></td>
    <td>100.0</td>
    <td>17.8</td>
    <td><b>89.1</b></td>
    <td><b>82.1</b></td>
    <td><b>85.1</b></td>
    <td><b>48.0</b></td>
    <td><b>68.0</b></td>
    <td><b>20.0</b></td>
    <td><b>53.0</b></td>
    <td><b>48.4</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+5.5</td>
    <td>-</td>
    <td>+1.5</td>
    <td>+2.5</td>
    <td>+1.8</td>
    <td>+2.9</td>
    <td>+5.5</td>
    <td>+8.0</td>
    <td>+2.5</td>
    <td>+3.0</td>
    <td>+3.5</td>
  </tr>
  <tr>
    <td>Alpaca - GPT4</td>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>41.9</td>
    <td>100.0</td>
    <td>32.0</td>
    <td>84.7</td>
    <td>80.4</td>
    <td>82.6</td>
    <td>39.5</td>
    <td>49.5</td>
    <td>14.5</td>
    <td>49.0</td>
    <td>38.1</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>46.3</td>
    <td>100.0</td>
    <td>27.6</td>
    <td>87.3</td>
    <td>84.1</td>
    <td>85.7</td>
    <td>42.0</td>
    <td>55.5</td>
    <td>16.5</td>
    <td>52.0</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>47.2</td>
    <td>100.0</td>
    <td>31.6</td>
    <td>86.7</td>
    <td>83.7</td>
    <td>85.2</td>
    <td>43.5</td>
    <td>59.0</td>
    <td>15.5</td>
    <td>51.5</td>
    <td>42.4</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>46.7</td>
    <td>100.0</td>
    <td>39.2</td>
    <td>84.4</td>
    <td>79.6</td>
    <td>82.0</td>
    <td>42.5</td>
    <td>58.0</td>
    <td>16.5</td>
    <td>52.0</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>46.9</td>
    <td>100.0</td>
    <td>41.1</td>
    <td>86.2</td>
    <td>81.1</td>
    <td>83.7</td>
    <td>43.5</td>
    <td>57.5</td>
    <td>17.0</td>
    <td>51.5</td>
    <td>42.4</td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>47.2</td>
    <td>100.0</td>
    <td><b>42.3</b></td>
    <td>87.0</td>
    <td>82.3</td>
    <td>84.7</td>
    <td>41.0</td>
    <td>56.0</td>
    <td>17.0</td>
    <td>52.0</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>NOVA - 5%</td>
    <td><b>50.5</b></td>
    <td>100.0</td>
    <td>39.8</td>
    <td><b>90.4</b></td>
    <td><b>85.2</b></td>
    <td><b>87.7</b></td>
    <td><b>45.4</b></td>
    <td><b>62.0</b></td>
    <td><b>20.5</b></td>
    <td><b>53.5</b></td>
    <td><b>48.3</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+3.8</td>
    <td>-</td>
    <td>+2.5</td>
    <td>+3.8</td>
    <td>+4.1</td>
    <td>+3.5</td>
    <td>+3.5</td>
    <td>+6.5</td>
    <td>+2.5</td>
    <td>+4.5</td>
    <td>+6.2</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>43.6</td>
    <td>100.0</td>
    <td>39.2</td>
    <td>86.5</td>
    <td>77.8</td>
    <td>82.2</td>
    <td>40.5</td>
    <td>56.0</td>
    <td>16.0</td>
    <td>49.5</td>
    <td>40.5</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td>45.9</td>
    <td>100.0</td>
    <td>38.0</td>
    <td>87.1</td>
    <td>78.3</td>
    <td>82.7</td>
    <td>43.0</td>
    <td>55.0</td>
    <td>15.5</td>
    <td>48.0</td>
    <td>40.4</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>46.8</td>
    <td>100.0</td>
    <td>35.7</td>
    <td>88.2</td>
    <td>80.1</td>
    <td>84.2</td>
    <td>41.5</td>
    <td>54.5</td>
    <td>16.5</td>
    <td>50.0</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>NOVA - 10%</td>
    <td><b>48.1</b></td>
    <td>100.0</td>
    <td>32.3</td>
    <td><b>90.6</b></td>
    <td><b>81.8</b></td>
    <td><b>86.2</b></td>
    <td><b>44.5</b></td>
    <td><b>59.0</b></td>
    <td><b>18.0</b></td>
    <td><b>51.0</b></td>
    <td><b>43.1</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+5.5</td>
    <td>-</td>
    <td>+1.8</td>
    <td>+3.9</td>
    <td>+1.4</td>
    <td>+3.6</td>
    <td>+4.0</td>
    <td>+7.0</td>
    <td>+2.5</td>
    <td>+6.0</td>
    <td>+6.8</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>42.9</td>
    <td>100.0</td>
    <td>32.2</td>
    <td>85.2</td>
    <td>80.3</td>
    <td>82.8</td>
    <td>46.0</td>
    <td>54.5</td>
    <td>15.0</td>
    <td>52.0</td>
    <td>41.9</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>44.6</td>
    <td>100.0</td>
    <td>33.6</td>
    <td>85.8</td>
    <td>81.5</td>
    <td>83.7</td>
    <td>45.5</td>
    <td>55.0</td>
    <td>15.0</td>
    <td>53.5</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>44.8</td>
    <td>100.0</td>
    <td>34.5</td>
    <td>86.1</td>
    <td>80.3</td>
    <td>83.4</td>
    <td>45.1</td>
    <td>52.0</td>
    <td>15.0</td>
    <td>53.1</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>NOVA - 15%</td>
    <td><b>46.9</b></td>
    <td>100.0</td>
    <td>32.1</td>
    <td><b>88.0</b></td>
    <td><b>82.5</b></td>
    <td><b>85.3</b></td>
    <td><b>49.5</b></td>
    <td><b>56.5</b></td>
    <td><b>18.5</b></td>
    <td><b>55.0</b></td>
    <td><b>44.9</b></td>
  </tr>
  <tr>
    <td>△ compared to Vanilla - 100%</td>
    <td>+5.0</td>
    <td>-</td>
    <td>+1.0</td>
    <td>+3.3</td>
    <td>+2.1</td>
    <td>+2.7</td>
    <td>+10.0</td>
    <td>+7.0</td>
    <td>+4.0</td>
    <td>+6.0</td>
    <td>+6.8</td>
  </tr>
</table>

Table 1: Results on three hallucination benchmarks. † indicates the factuality hallucination benchmark. ‡ indicates the faithfulness hallucination benchmark. We conduct the experiments based on LLaMA-3-8B.

- **100%** fine-tunes the model on the whole instruction dataset; (2) Instruction Data Filtering Methods: **IFD** (Li et al., 2024a) proposes instruction-following difficulty to select a subset of instruction data. **CaR** (Ge et al., 2024) simultaneously considers the data quality and diversity by introducing two scoring methods. **Nuggets** (Li et al., 2024b) focuses on selecting high-quality data by identifying samples that notably boost the performance of different tasks after being learned as one-shot instances; (3) RL-based Methods: **FLAME-DPO<sup>fact</sup>** (Lin et al., 2024b) introduces atomic fact decomposition and retrieval augmented claim verification to construct preference data and apply DPO. **SELF-EVAL** (Zhang et al., 2024b) leverages the self-evaluation capability of LLMs and employs GPT-3.5 to create preference data, aligning the LLM with DPO. We apply these RL-based methods after tuning LLMs on the whole instruction dataset.

**Implementation Details.** Our main experiments are conducted on LLaMA-3-8B and LLaMA-3-70B (Grattafiori et al., 2024). More implementation details are shown in Appendix B, e.g., the training of quality reward model and hyperparameters.

**4.2 Main Results**

**NOVA Significantly Reduces Hallucinations.** As shown in Table 1, NOVA shows consistent and significant improvements on three hallucination benchmarks measuring factuality and faithfulness. Compared to indiscriminately using the whole instruction dataset (i.e., Vanilla - 100%), using samples selected by NOVA to train LLMs can improve **3.5-8.6%** on BioGEN, **2.0-5.1%** on LongFact, and **4.9-8.5%** on FollowRAG-Faithfulness. This is because NOVA effectively filters out the unfamiliar instruction data and avoids training LLMs on these data thereby reducing the hallucinations. Compared to instruction data filtering methods that focus on data quality, like IFD, our method consistently improves the performance across different selected sample ratios (5-15%) on three benchmarks. Meanwhile, these data selected by quality-focused methods may present unfamiliar knowledge to the LLM and encourage hallucinations on LongFact. On the contrary, NOVA aims to identify the sam-
<table>
  <tr>
    <th>Model</th>
    <th>MT-Bench</th>
    <th>FollowRAG-Instruction</th>
  </tr>
  <tr>
    <td colspan="3" style="background-color:#f2cc8f;">Alpaca</td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>51.9</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>46.7</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>48.3</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>60.1</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>56.6</td>
    <td><b>41.4</b></td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>60.0</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td><b>NOVA - 5%</b></td>
    <td><b>60.5</b></td>
    <td>39.1</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+6.6</td>
    <td>+6.4</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>57.2</td>
    <td>40.4</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td><b>58.3</b></td>
    <td><b>42.3</b></td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>58.2</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td><b>NOVA - 10%</b></td>
    <td><b>56.6</b></td>
    <td>38.8</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+4.7</td>
    <td>+0.1</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>56.0</td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>57.4</td>
    <td><b>41.0</b></td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>57.0</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td><b>NOVA - 15%</b></td>
    <td>57.2</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+5.5</td>
    <td>+1.4</td>
  </tr>
  <tr>
    <td colspan="3" style="background-color:#f2cc8f;">Alpaca-GPT4</td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>64.3</td>
    <td>36.9</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>56.2</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>53.1</td>
    <td>36.5</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>65.0</td>
    <td>37.0</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>65.4</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>66.2</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td><b>NOVA - 5%</b></td>
    <td>64.6</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+0.3</td>
    <td>+0.9</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>65.0</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td>65.8</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td><b>67.5</b></td>
    <td>38.0</td>
  </tr>
  <tr>
    <td><b>NOVA - 10%</b></td>
    <td>64.6</td>
    <td><b>39.1</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+6.3</td>
    <td>+2.1</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>62.3</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>61.1</td>
    <td><b>38.1</b></td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>66.5</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td><b>NOVA - 15%</b></td>
    <td>64.5</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+0.2</td>
    <td>+0.5</td>
  </tr>
</table>

Table 2: Results on two instruction-following benchmarks implemented on LLaMA-3-8B.

![Bar chart showing perplexity scores for various models](page_1012_120_410_312.png)

Figure 3: Average perplexity score of 15 samples with the lowest scores for each model from LongFact-Objects. Models are trained on Alpaca-GPT4.

<table>
  <tr>
    <th>Model</th>
    <th>BioGEN</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <td>NOVA - 5% - 70B</td>
    <td>60.9</td>
    <td>74.3</td>
  </tr>
  <tr>
    <td>-w/o. Data Filtering</td>
    <td>53.7</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td><b>NOVA-5% -8B</b></td>
    <td>50.5</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td>-w/o. Data Filtering</td>
    <td>41.9</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>-w/o. ICP</td>
    <td>47.6</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>-w/o. SEI</td>
    <td>48.2</td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>-w/o. Quality RM</td>
    <td>55.6</td>
    <td>48.6</td>
  </tr>
  <tr>
    <td>-w/o. ICP & SEI</td>
    <td>43.7</td>
    <td>65.2</td>
  </tr>
</table>

Table 3: Results of ablation and scalability study. We report FactScore results on BioGEN. Models are trained on Alpaca-GPT4. RM represents the reward model.

samples that align well with LLM’s knowledge, helping the LLM to hallucinate less. NOVA also achieves better performance than RL-based methods without introducing additional preference data. These findings underline the effectiveness of our method in aligning LLMs to hallucinate less.

NOVA Maintains a Good Balance between Following Instructions and Reducing Hallucinations. As shown in Table 2, NOVA achieves a better instruction-following ability compared to vanilla tuning methods, especially when the LLM is trained on Alpaca. It shows that NOVA can effectively align LLMs to follow instructions. In some cases, our method surpasses data filtering methods that enhance instruction-following ability, demonstrating its effectiveness in identifying suitable data for LLMs. Unlike RL-based methods that weaken the model’s instruction-following ability, our method shows superior instruction-following ability while greatly reducing hallucinations.

NOVA Mitigates Overconfidence Phenomenon. We select 15 samples with the lowest scores for each model from LongFact-Objects and calculate its average perplexity on these samples. We find that NOVA generates a high perplexity score (i.e., low sentence-level confidence score) on these bad cases as shown in Figure 3, showing that NOVA mitigates overconfidence in these false statements.

4.3 Analysis

Ablation Study. We conduct the ablation study in Table 3. We can find that the proposed ICP and SEI can both help LLMs to reduce hallucinations. Also, considering only familiarity (i.e., -w/o. Quality RM) ignores other important characteristics of instruction data and limits the instruction-following ability of LLMs. Thus, even if considering familiarity alone would greatly reduce hallucinations, it is still necessary to introduce a quality reward model to maintain a good balance between following instructions and reducing hallucinations.

Scalability Study. We implement our method on the LLaMA-3-70B in Table 3 to explore whether NOVA can fit in larger LLMs. Results indicate that NOVA scales effectively to larger-scale models.

Case Study. We conduct a case study in Table 4 to visually show the advantages of NOVA. Compared to using the whole training data, our method ensures the statements are correct and comprehensive, and the generated text is fluent and natural.
Instruction: Who is the character Geralt of Rivia from The Witcher series?

Vanilla - 100%: Geralt of Rivia is a character and the main protagonist of the fantasy novel series “The Witcher” written by Andrzej Sapkowski (✓). He is a witcher, a rare human mutation born with powers (✓) to fight against the monsters that threaten the world of the living (✓). Geralt is known for his intelligence (✓), skill in combat (✓), and his collection of magical artifacts, particularly his silver sword (✓). Throughout the series, Geralt faces many dangers and battles as he journeys across the Continent, accompanied by his friends and lovers, in a quest to protect the world and the people he cares about (✓).

NOVA-5%: Geralt of Rivia is a character from The Witcher series, a fantasy franchise that includes a series of novels, short stories, and video games (✓). He is a witcher, a member of a guild of professional monster hunters, and his role is to protect the innocent from supernatural threats (✓). Geralt is a complex and flawed character with a strong sense of honor and duty, and a deep understanding of the world around him (✓). He is often portrayed as a loner, but he is also capable of forming close relationships with others, particularly with those who share his beliefs and values (✓).

<table>
  <tr>
    <th>Model</th>
    <th>BioGEN</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <td>NOVA-5% - Alpaca-GPT4</td>
    <td>50.5</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td>&#9733; w/o ICP</td>
    <td>48.4</td>
    <td>62.2</td>
  </tr>
  <tr>
    <td>&#9733; Confidence Score (Perplexity)</td>
    <td>47.9</td>
    <td>61.5</td>
  </tr>
  <tr>
    <td>&#9733; Lexical Similarity (Rouge-L)</td>
    <td>49.8</td>
    <td>63.9</td>
  </tr>
  <tr>
    <td>&#9733; Using Embedding Model</td>
    <td>48.5</td>
    <td>63.2</td>
  </tr>
  <tr>
    <td>&#9733; w/o SEI</td>
    <td>47.8</td>
    <td>60.2</td>
  </tr>
  <tr>
    <td>&#9733; K-means Clustering via Internal States</td>
    <td>48.5</td>
    <td>63.2</td>
  </tr>
  <tr>
    <td>&#9733; K-means Clustering via Embedding Model</td>
    <td>47.3</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>&#9733; Voting without Semantic Clustering</td>
    <td>47.3</td>
    <td>60.8</td>
  </tr>
</table>

Table 4: Case study from LongFact-Objects. We highlight the statements that share the same semantics using the same color. Models are trained on Alpaca-GPT4.

<table>
  <tr>
    <th></th>
    <th>Ours Wins</th>
    <th>Tie</th>
    <th>Vanilla Wins</th>
  </tr>
  <tr>
    <th>Factuality</th>
    <td>0.62</td>
    <td>0.1</td>
    <td>0.28</td>
  </tr>
  <tr>
    <th>Naturalness</th>
    <td>0.38</td>
    <td>0.34</td>
    <td>0.28</td>
  </tr>
  <tr>
    <th>Relevance</th>
    <td>0.48</td>
    <td>0.26</td>
    <td>0.26</td>
  </tr>
  <tr>
    <th>Helpfulness</th>
    <td>0.4</td>
    <td>0.28</td>
    <td>0.32</td>
  </tr>
</table>

Figure 4: Human evaluation across four key dimensions. The models are trained on Alpaca-GPT4.

Table 5: Evaluation results of NOVA that employ various methods for measuring the LLM’s familiarity. We report FactScore results on BioGEN.

Variant Methods Testing. As shown in Table 5, we further explore the variant methods in measuring the LLM’s familiarity. For ICP, we separately replace it with sentence-level confidence (Perplexity) and lexical metrics (Rouge-L). Specifically, we use the average perplexity score of generated responses to represent sentence-level confidence and use the average Rouge-L score between each pair of two generated responses as lexical metrics. However, these straightforward strategies neglect highly concentrated semantic information within the internal states, and thus fail to capture the fine-grained differences between responses and limit the final performance. We also explore the effectiveness of an advanced embedding model, we use TEXT-EMBEDDING-3-LARGE\(^2\) from OpenAI and set the dimension as 4096. We find that using the internal states achieves better performance, showing the effectiveness of our method. This is because internal states may reflect more dense and fine-grained information from LLM itself that may have been lost in the decoding phase of the responses. For SEI, we explore whether using k-means clustering based on internal states computed as ICP and sentence embedding from TEXT-EMBEDDING-3-LARGE can

identify suitable semantic clusters. We can find that our method achieves better performance because the k-means algorithm is not based on semantic equivalence to get the clusters. Also, the internal states of LLMs cannot efficiently represent the target response, as this response is manually labeled or generated by other advanced LLMs instead of generated by the LLM itself. We also find that simply voting based on the textual contents instead of semantic clustering limits the final performance, as these responses are often free-form and can have the same meaning in different ways.

Discussion. We conduct the parameter study to test the robustness of our method in Appendix C. We also conduct a transferability study in Appendix D and find NOVA can fit in other LLMs. We further explore the design of our method in Appendix E and find our design is effective. We conduct a case study in Appendix G to qualitatively show the difference between samples with different scores.

Human Evaluation. We conduct a human evaluation on the 50 generated biographies from BioGEN across four key dimensions: factuality, helpfulness, relevance, and naturalness. For each comparison, three options are given (Ours Wins, Tie, and Vanilla Fine-tuning Wins) and the majority voting determines the final result. Figure 4 shows that our method significantly reduces hallucinations and effectively follows instructions with high-quality responses. Details can be found in Appendix F.

5 Conclusion

In this paper, we introduce NOVA, a novel framework designed to identify high-quality data that aligns well with the LLM’s learned knowledge to reduce hallucination. NOVA includes Internal Consistency Probing and Semantic Equivalence Identification, which are designed to separately measure the LLM’s familiarity with the given instruction
and target response, then prevent the model from being trained on unfamiliar data, thereby reducing hallucinations. Lastly, we introduce an expert-aligned reward model, considering characteristics beyond just familiarity to enhance data quality. By considering data quality and avoiding unfamiliar data, we can use the selected data to effectively align LLMs to follow instructions and hallucinate less in the instruction tuning stage. Experiments and analysis show the effectiveness of NOVA.

Limitations

Although empirical experiments have confirmed the effectiveness of the proposed NOVA, two major limitations remain. Firstly, our proposed method requires LLMs to generate multiple responses for the given instruction, which introduces additional execution time. However, it is worth noting that this additional execution time is used to perform offline data filtering, our proposed method does not introduce additional time overhead in the inference phase. Additionally, NOVA is primarily used for single-turn instruction data filtering, thus exploring its application in multi-turn scenarios presents an attractive direction for future research.

Acknowledgements

We would like to thank the anonymous reviewers for their thoughtful and constructive comments. This work is supported by the National Science and Technology Major Project (2020AAA0106502), the National Natural Science Foundation of China (No. T2341003), and a grant from the Guoqiang Institute, Tsinghua University.

References

Anthropic. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.

Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it’s lying. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. 2024. Instruction mining: Instruction data selection for tuning large language models. Preprint, arXiv:2307.06290.

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024a. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations.

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024b. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations.

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701.

Yi Cheng, Xiao Liang, Yeyun Gong, Wen Xiao, Song Wang, Yuji Zhang, Wenjun Hou, Kaishuai Xu, Wenge Liu, Wenjie Li, Jian Jiao, Qi Chen, Peng Cheng, and Wayne Xiong. 2024. Integrative decoding: Improve factuality via implicit self-consistency. Preprint, arXiv:2410.01556.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanning Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRA_FEEDBACK: Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning.

Guanting Dong, Xiaoshuai Song, Yutao Zhu, Runqi Qiao, Zhicheng Dou, and Ji-Rong Wen. 2024. Toward general instruction-following alignment for retrieval-augmented generation. Preprint, arXiv:2410.09584.

Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Hongxia Ma, Li Zhang, Hao Yang, and Tong Xiao. 2024. Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation. arXiv preprint arXiv:2402.18191.

Zorik Gekhman, Gal Yona, Roei Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning LLMs on new knowledge encourage hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7765–7784, Miami, Florida, USA. Association for Computational Linguistics.

Aaron Grattafiore, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esioibu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy,
Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Michelon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junting Jia, Kalyan Vasudean Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimopoulkeli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam-badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathiy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girhard, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbally, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seo-hyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Shan-ran Narang, Sharath Raparthi, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Van-denheide, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Colot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheashua, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujwial Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gouguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyu Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xi-aofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xin-feng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Couderet, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajineld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenber, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bhamrane, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Chings-Hsian Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcueta, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippo Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Gunja Lakshminarayanan, Hakan Inan, Hamid Shojanazari, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molbyog, Igor Tufanon, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Grovesh, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michael Valko, Michelle Restrepo, Mihir Patel, Mik Vyatkovsk, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Prithish Vuyraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky
Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsey, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzouk Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihaleaescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojuan Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhao Duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. Just Accepted.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2024. Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, low-quality model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4439–4454, Mexico City, Mexico. Association for Computational Linguistics.

Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. 2024. Unfamiliar finetuning examples control how language models hallucinate. Preprint, arXiv:2403.05612.

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024a. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7595–7628, Mexico City, Mexico. Association for Computational Linguistics.

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. 2024b. One shot learning as instruction data prospector for large language models. Preprint, arXiv:2312.10302.

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khayathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2024a. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations.

Sheng-Chieh Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen tau Yih, and Xilun Chen. 2024b. FLAME : Factuality-aware alignment for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024c. Generating with confidence: Uncertainty quantification for black-box large language models. Trans. Mach. Learn. Res., 2024.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024a. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations.

Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, Hongxia Ma, Li Zhang, Hao Yang, and Yanfei Jiang. 2024b. Coachlm: Automatic instruction revisions improve the data quality in LLM instruction tuning. In 40th IEEE International Conference on Data Engineering, ICDE 2024, Utrecht, The Netherlands, May 13-16, 2024, pages 5184–5197. IEEE.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActscore: Fine-grained atomic evaluation of factual precision in long form text generation. In The 2023 Conference on Empirical Methods in Natural Language Processing.

OpenAI. 2022. large-scale generative pre-training model for conversation. OpenAI blog.

OpenAI. 2023a. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

OpenAI. 2023b. Openai: Gpt-4.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS, pages 27730–27744.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. Preprint, arXiv:2304.03277.

Rafael Rafaillov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.

Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. Preprint, arXiv:2309.05922.

Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations.

Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. 2023. SpokenWOZ: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, and Maosong Sun. 2024. Gateau: Selecting influential sample for long context alignment. arXiv preprint arXiv:2410.15633.

Shuzheng Si, Haozhe Zhao, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Bofei Gao, Kangyang Luo, Wenhao Li, Yufei Huang, Gang Chen, et al. 2025. Teaching large language models to maintain contextual faithfulness via synthetic tasks and reinforcement learning. arXiv preprint arXiv:2505.16483.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Wen tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Annual Meeting of the Association for Computational Linguistics.

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023b. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024a. Long-form factuality in large language models.

Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024b. Long-form factuality in large language models. Preprint, arXiv:2403.18802.

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML).

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhiliao Fan. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

Qi Zhang, Yiming Zhang, Haobo Wang, and Junbo Zhao. 2024a. Recost: External knowledge guided data-efficient instruction tuning. Preprint, arXiv:2402.17355.

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024b. Self-alignment for factuality: Mitigating hallucinations in LLMs via self-evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1965, Bangkok, Thailand. Association for Computational Linguistics.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206.
Appendix

A Evaluation

In this section, we will detail the benchmarks and evaluation metrics.

BioGEN. (Factuality) This benchmark requires generating short biographies for particular people entities, with a total of 500 samples. The task of generating people biographies is effective, because generations consist of verifiable statements rather than debatable or subjective ones, and the scope is broad (i.e., covering diverse nationalities, professions, and levels of rarity). To evaluate each generated response, we follow the FactScore procedure to extract the number of correct and incorrect facts. Following Min et al. (2023), we first employ GPT-3.5-Turbo-0125 to break a generation into a series of atomic facts and utilize GPT-3.5-Turbo-0125 to compute the percentage of atomic facts supported by a reliable knowledge source. The percentage of the correct statements (% FactScore), the number of generated statements (# Facts), and the ratio of generations that do not abstain from responding (% Respond) are adopted as the evaluation metrics.

LongFact. (Factuality) LongFact requests detailed descriptions for a queried entity and expects a document-level response that is typically very long, often exceeding a thousand tokens. Specifically, LongFact consists of two subtasks: LongFact-Concepts and LongFact-Objects, separated based on whether the questions ask about concepts or objects. Following Cheng et al. (2024), we use 120 samples of each task for evaluation. The evaluation process is similar to BioGEN. We employ GPT-3.5-Turbo-0125 and report the FactScore of LongFact-Concepts and LongFact-Objects, termed as % Concepts and % Objects.

FollowRAG. (Faithfulness and Instruction Following) FollowRAG aims to assess the model’s ability to follow user instructions in complex multi-document contexts, covering 22 fine-grained atomic instructions across 6 categories. The queries in FollowRAG are sourced from 4 QA datasets across NaturalQA (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018), and WebQSP (tau Yih et al., 2016). It collects and verifies definitions and examples of atomic instructions using rules (e.g., code), excluding those irrelevant to retrieval-augmented generation (RAG) scenarios. FollowRAG identifies 22 types of instruction constraints, encompassing language, length, structure, and keywords. Thus, it is suitable to use FollowRAG to evaluate the model’s ability to follow user instructions. Utilizing the verifiable nature of designed atomic instructions, FollowRAG automates the verification of the model’s adherence to each instruction through code validation. We calculate the average pass rate for each atomic instruction across all samples to determine the instruction-following score and name this task as FollowRAG-Instruction. Also, FollowRAG provides retrieved passages as contextual information to evaluate the model’s faithfulness. We name this task as FollowRAG-Faithfulness. Under new instruction constraints, the model’s target output differs from the gold answers in the original QA dataset, rendering traditional metrics like EM ineffective. Following Dong et al. (2024), we use the original gold answers as a reference and utilize GPT-4o-2024-05-13 to evaluate whether the model’s outputs address the questions. The scoring criteria are as follows: Completely correct (1 point), Partially correct (0.5 points), Completely incorrect (0 points). The average score of all samples is taken as the final score for FollowRAG-Faithfulness.

MT-Bench. (Instruction Following) MT-Bench is a benchmark consisting of 80 questions, designed to test instruction-following ability, covering common use cases and challenging questions. It is also carefully constructed to differentiate chatbots based on their core capabilities, including writing, role-play, extraction, reasoning, math, coding, STEM knowledge, and social science. For evaluation, MT-Bench prompts GPT-4 to act as judges and assess the quality of the models’ responses. For each turn, GPT-4 will give a score on a scale of 10. Notably, since we only fine-tune on single-turn instruction data (e.g., Alpaca and Alpaca-GPT4), the evaluation is restricted to Turn 1 of MTBench, similar to previous studies (Li et al., 2024b).

B Implementation Details

Hyperparameters and Devices. We use Adam optimizer (Kingma and Ba, 2017) to train our model, with a \( 2 \times 10^{-5} \) learning rate and a batch size of 16, steers the training across three epochs. We set the maximum input length for the models to 1024. To get the generated initial responses for knowledge estimation, we set the temperature as 0.7 and set hyperparameter \( K \) as 10 to generate 10 responses for the given instruction \( q \). We conduct
our experiments on NVIDIA A800 80G GPUs with DeepSpeed+ZeRO3 and BF16.

Training of NLI Model. Natural language inference (NLI) is a well-studied task in the NLP community. We employ a well-trained NLI model DeBERTa-large-mnli3 (He et al., 2021) (0.3B) as our model to conduct the experiments and report the results. DeBERTa-large-mnli is the DeBERTa large model fine-tuned with multi-genre natural language inference (MNLI) corpus (Williams et al., 2018), which is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information. DeBERTa-large-mnli shows advanced performance in various NLI benchmarks e.g., 91.5% accuracy on MNLI test set.

Traning of Quality Reward Model. Our training data is derived from an expert-revised dataset (Liu et al., 2024b), which consists of 3,751 instruction pairs from Alpaca refined by linguistic experts to enhance fluency, accuracy, and semantic coherence between instructions and responses. Meanwhile, Liu et al. (2024b) employs the edit distance metric (i.e., Levenshtein distance) to assess the quality of the original instruction pair and revised instruction pair. Thus, we can treat this edit distance metric as the target reward value and use the point-wise loss function to train the reward model. Specifically, following Ge et al. (2024), we concatenate instruction pairs as text inputs and use the given reward value in the dataset as the target outputs. We use the average pooling strategy and introduce the additional feed-forward layer to transform the hidden states of the model into a scalar. Then we use Mean Squared Error as the loss function to train the reward model. We select DeBERTa-large (He et al., 2021) (0.3B) as our model. We use Adam optimizer to train our model, with a \(1.5 \times 10^{-5}\) learning rate and a batch size of 8. We train our model on a single NVIDIA A800.

Prompt Template. We use the prompt template from Alpaca (Taori et al., 2023). We keep the same template in training and inference.

C Parameter Study

We explore the effects of two important hyperparameters in our method: the number of generated responses \(K\) and the temperature \(T\) during the response generation. As shown in Figure 5, increasing the number of generated responses improves the performance of our method, but when the number of generated responses is greater than 10, the performance will be stable. Therefore, we empirically recommend setting the number of generated responses \(K\) to 10, which makes our method effective and efficient. For the temperature \(T\), we find that the performance of the model improves as long as the temperature \(T\) is chosen wisely and not at an extreme value (e.g., 0, as this would result in multiple generated responses that are exactly the same). We recommend that the temperature take a moderate value, as this ensures both that there is diversity in the responses generated and that the generated responses do indeed match the model’s perceptions (rather than being too random). Overall, our method NOVA is robust to these hyperparameters, making our method easy to follow.

![Line graph showing FactScore on BioGEN vs. The Number of Generated Responses for Alpaca and Alpaca-GPT4](page_1012_120_377_273.png)

Figure 5: FactScore results on BioGEN with the different number of generated responses \(K\). We conduct the experiments based on LLaMA-3-8B.

<table>
  <tr>
    <th>Model</th>
    <th>Dataset</th>
    <th>BioGEN</th>
  </tr>
  <tr>
    <td>NOVA</td>
    <td>Alpaca</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>\(-T = 0\)</td>
    <td>Alpaca</td>
    <td>43.2</td>
  </tr>
  <tr>
    <td>\(-T = 0.2\)</td>
    <td>Alpaca</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>\(-T = 0.7\) (Ours)</td>
    <td>Alpaca</td>
    <td><b>50.3</b></td>
  </tr>
  <tr>
    <td>\(-T = 1.0\)</td>
    <td>Alpaca</td>
    <td>50.1</td>
  </tr>
  <tr>
    <td>\(-T = 1.3\)</td>
    <td>Alpaca</td>
    <td>49.7</td>
  </tr>
  <tr>
    <td>NOVA</td>
    <td>Alpaca-GPT4</td>
    <td><b>50.5</b></td>
  </tr>
  <tr>
    <td>\(-T = 0\)</td>
    <td>Alpaca-GPT4</td>
    <td>43.6</td>
  </tr>
  <tr>
    <td>\(-T = 0.2\)</td>
    <td>Alpaca-GPT4</td>
    <td>48.9</td>
  </tr>
  <tr>
    <td>\(-T = 0.7\) (Ours)</td>
    <td>Alpaca-GPT4</td>
    <td><b>50.5</b></td>
  </tr>
  <tr>
    <td>\(-T = 1.0\)</td>
    <td>Alpaca-GPT4</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>\(-T = 1.3\)</td>
    <td>Alpaca-GPT4</td>
    <td>49.5</td>
  </tr>
</table>

Table 6: FactScore results on BioGEN with different temperature \(T\) during the response generation. We conduct the experiments on LLaMA-3-8B and use 5% selected instruction data from different datasets.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">BioGEN†</th>
    <th colspan="4">LongFact†</th>
    <th colspan="4">FollowRAG - Faithfulness‡</th>
  </tr>
  <tr>
    <th>FactScore</th>
    <th>Respond</th>
    <th>Facts</th>
    <th>Objects</th>
    <th>Concepts</th>
    <th>Avg.</th>
    <th>NaturalQA</th>
    <th>TriviaQA</th>
    <th>HotpotQA</th>
    <th>WebQSP</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td colspan="12">LLaMA</td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>38.6</td>
    <td>100.0</td>
    <td>16.6</td>
    <td>84.3</td>
    <td>78.2</td>
    <td>81.3</td>
    <td>37.5</td>
    <td>50.5</td>
    <td>16.0</td>
    <td>47.5</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>41.2</td>
    <td>100.0</td>
    <td>14.8</td>
    <td>86.7</td>
    <td>81.2</td>
    <td>84.0</td>
    <td>41.5</td>
    <td>55.0</td>
    <td>21.5</td>
    <td>52.5</td>
    <td>42.6</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>41.8</td>
    <td>100.0</td>
    <td>15.7</td>
    <td>87.0</td>
    <td>80.8</td>
    <td>83.9</td>
    <td>42.5</td>
    <td>56.5</td>
    <td>22.5</td>
    <td>53.5</td>
    <td>43.8</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>40.2</td>
    <td>100.0</td>
    <td>20.1</td>
    <td>83.2</td>
    <td>80.4</td>
    <td>81.8</td>
    <td>38.0</td>
    <td>53.5</td>
    <td>18.5</td>
    <td>49.0</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>39.6</td>
    <td>100.0</td>
    <td>18.2</td>
    <td>85.9</td>
    <td>80.1</td>
    <td>83.0</td>
    <td>38.0</td>
    <td>53.0</td>
    <td>19.0</td>
    <td>50.5</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>39.3</td>
    <td>100.0</td>
    <td>19.4</td>
    <td>85.1</td>
    <td>77.3</td>
    <td>81.2</td>
    <td>39.5</td>
    <td>54.5</td>
    <td>20.0</td>
    <td>50.0</td>
    <td>41.0</td>
  </tr>
  <tr>
    <td><b>NOVA - 5%</b></td>
    <td><b>43.6</b></td>
    <td><b>100.0</b></td>
    <td><b>21.5</b></td>
    <td><b>88.1</b></td>
    <td><b>82.5</b></td>
    <td><b>85.3</b></td>
    <td><b>44.5</b></td>
    <td><b>58.5</b></td>
    <td><b>24.0</b></td>
    <td><b>55.5</b></td>
    <td><b>45.6</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+5.0</td>
    <td>-</td>
    <td>+4.9</td>
    <td>+3.8</td>
    <td>+4.3</td>
    <td>+4.1</td>
    <td>+7.0</td>
    <td>+8.0</td>
    <td>+8.0</td>
    <td>+7.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>40.7</td>
    <td>100.0</td>
    <td>19.2</td>
    <td>85.2</td>
    <td>80.3</td>
    <td>82.8</td>
    <td>40.0</td>
    <td>54.5</td>
    <td>20.0</td>
    <td>51.0</td>
    <td>41.4</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td>40.3</td>
    <td>100.0</td>
    <td>21.1</td>
    <td>83.4</td>
    <td>79.2</td>
    <td>81.3</td>
    <td>41.0</td>
    <td>52.0</td>
    <td>18.0</td>
    <td>49.5</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>41.0</td>
    <td>100.0</td>
    <td>18.8</td>
    <td>84.2</td>
    <td>78.6</td>
    <td>81.4</td>
    <td>39.5</td>
    <td>53.0</td>
    <td>17.5</td>
    <td>51.0</td>
    <td>40.3</td>
  </tr>
  <tr>
    <td><b>NOVA - 10%</b></td>
    <td><b>43.2</b></td>
    <td><b>100.0</b></td>
    <td><b>20.7</b></td>
    <td><b>87.6</b></td>
    <td><b>83.2</b></td>
    <td><b>85.4</b></td>
    <td><b>49.5</b></td>
    <td><b>59.5</b></td>
    <td><b>22.5</b></td>
    <td><b>53.0</b></td>
    <td><b>44.6</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+2.6</td>
    <td>-</td>
    <td>+4.1</td>
    <td>+3.3</td>
    <td>+3.0</td>
    <td>+4.2</td>
    <td>+9.0</td>
    <td>+16.5</td>
    <td>+5.5</td>
    <td>+7.6</td>
    <td>-</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>39.2</td>
    <td>100.0</td>
    <td>18.7</td>
    <td>86.1</td>
    <td>81.1</td>
    <td>83.6</td>
    <td>39.6</td>
    <td>52.0</td>
    <td>17.5</td>
    <td>49.5</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>40.2</td>
    <td>100.0</td>
    <td>18.5</td>
    <td>84.2</td>
    <td>80.4</td>
    <td>82.3</td>
    <td>38.0</td>
    <td>51.5</td>
    <td>17.0</td>
    <td>48.0</td>
    <td>38.6</td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>40.9</td>
    <td>100.0</td>
    <td>18.1</td>
    <td>83.3</td>
    <td>80.0</td>
    <td>81.7</td>
    <td>40.0</td>
    <td>52.5</td>
    <td>15.5</td>
    <td>50.5</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td><b>NOVA - 15%</b></td>
    <td><b>44.1</b></td>
    <td><b>100.0</b></td>
    <td><b>19.4</b></td>
    <td><b>89.6</b></td>
    <td><b>83.7</b></td>
    <td><b>86.7</b></td>
    <td><b>45.5</b></td>
    <td><b>56.5</b></td>
    <td><b>23.5</b></td>
    <td><b>54.5</b></td>
    <td><b>44.3</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+5.5</td>
    <td>-</td>
    <td>+2.8</td>
    <td>+5.3</td>
    <td>+3.5</td>
    <td>+5.4</td>
    <td>+5.0</td>
    <td>+16.0</td>
    <td>+7.5</td>
    <td>+6.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="12">Qwen-2</td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>40.3</td>
    <td>100.0</td>
    <td>17.3</td>
    <td>83.4</td>
    <td>80.2</td>
    <td>81.8</td>
    <td>39.5</td>
    <td>57.5</td>
    <td>18.5</td>
    <td>49.0</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fact</sup></td>
    <td>47.1</td>
    <td>100.0</td>
    <td>16.9</td>
    <td>87.8</td>
    <td>82.7</td>
    <td>85.3</td>
    <td>44.5</td>
    <td>58.0</td>
    <td>20.5</td>
    <td>53.0</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>46.8</td>
    <td>100.0</td>
    <td>14.2</td>
    <td>88.2</td>
    <td>81.6</td>
    <td>84.9</td>
    <td>43.5</td>
    <td>59.0</td>
    <td>21.0</td>
    <td>53.0</td>
    <td>44.1</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>44.2</td>
    <td>100.0</td>
    <td>16.5</td>
    <td>85.2</td>
    <td>81.2</td>
    <td>83.2</td>
    <td>42.5</td>
    <td>56.5</td>
    <td>20.5</td>
    <td>53.5</td>
    <td>43.3</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td>45.7</td>
    <td>100.0</td>
    <td>18.6</td>
    <td>84.1</td>
    <td>81.5</td>
    <td>82.8</td>
    <td>44.5</td>
    <td>55.5</td>
    <td>21.0</td>
    <td>52.0</td>
    <td>43.3</td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>46.6</td>
    <td>100.0</td>
    <td>18.3</td>
    <td>84.7</td>
    <td>81.8</td>
    <td>82.9</td>
    <td>43.0</td>
    <td>57.5</td>
    <td>21.5</td>
    <td>52.5</td>
    <td>43.6</td>
  </tr>
  <tr>
    <td><b>NOVA - 5%</b></td>
    <td><b>49.1</b></td>
    <td><b>100.0</b></td>
    <td><b>18.3</b></td>
    <td><b>90.2</b></td>
    <td><b>83.2</b></td>
    <td><b>86.7</b></td>
    <td><b>46.0</b></td>
    <td><b>59.6</b></td>
    <td><b>23.5</b></td>
    <td><b>55.5</b></td>
    <td><b>46.1</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+4.8</td>
    <td>-</td>
    <td>+2.1</td>
    <td>+6.8</td>
    <td>+3.0</td>
    <td>+4.5</td>
    <td>+5.5</td>
    <td>+2.1</td>
    <td>+5.0</td>
    <td>+5.5</td>
    <td>-</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>44.5</td>
    <td>100.0</td>
    <td>17.8</td>
    <td>84.2</td>
    <td>80.5</td>
    <td>82.4</td>
    <td>41.5</td>
    <td>59.5</td>
    <td>19.5</td>
    <td>51.0</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td>45.2</td>
    <td>100.0</td>
    <td>20.3</td>
    <td>84.5</td>
    <td>79.8</td>
    <td>82.2</td>
    <td>42.5</td>
    <td>60.0</td>
    <td>18.5</td>
    <td>53.0</td>
    <td>43.5</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>46.1</td>
    <td>100.0</td>
    <td>23.5</td>
    <td>85.2</td>
    <td>79.7</td>
    <td>82.5</td>
    <td>42.0</td>
    <td>60.0</td>
    <td>20.0</td>
    <td>51.5</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td><b>NOVA - 10%</b></td>
    <td><b>47.5</b></td>
    <td><b>100.0</b></td>
    <td><b>18.6</b></td>
    <td><b>89.6</b></td>
    <td><b>83.5</b></td>
    <td><b>86.6</b></td>
    <td><b>45.0</b></td>
    <td><b>62.0</b></td>
    <td><b>21.5</b></td>
    <td><b>53.5</b></td>
    <td><b>45.5</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+3.2</td>
    <td>-</td>
    <td>+2.1</td>
    <td>+5.2</td>
    <td>+3.0</td>
    <td>+4.4</td>
    <td>+3.5</td>
    <td>+12.0</td>
    <td>+3.0</td>
    <td>+4.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>43.7</td>
    <td>100.0</td>
    <td>19.2</td>
    <td>82.5</td>
    <td>79.5</td>
    <td>81.0</td>
    <td>42.0</td>
    <td>61.5</td>
    <td>18.5</td>
    <td>52.0</td>
    <td>43.5</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td>44.8</td>
    <td>100.0</td>
    <td>20.8</td>
    <td>81.2</td>
    <td>81.3</td>
    <td>81.3</td>
    <td>43.0</td>
    <td>62.5</td>
    <td>19.5</td>
    <td>53.0</td>
    <td>44.5</td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>45.7</td>
    <td>100.0</td>
    <td>21.7</td>
    <td>80.8</td>
    <td>80.5</td>
    <td>80.5</td>
    <td>40.5</td>
    <td>62.5</td>
    <td>20.5</td>
    <td>52.5</td>
    <td>43.9</td>
  </tr>
  <tr>
    <td><b>NOVA - 15%</b></td>
    <td><b>47.2</b></td>
    <td><b>100.0</b></td>
    <td><b>19.3</b></td>
    <td><b>88.8</b></td>
    <td><b>82.9</b></td>
    <td><b>85.9</b></td>
    <td><b>44.5</b></td>
    <td><b>64.5</b></td>
    <td><b>22.0</b></td>
    <td><b>54.0</b></td>
    <td><b>46.3</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>†</sup></td>
    <td>+3.5</td>
    <td>-</td>
    <td>+2.0</td>
    <td>+7.6</td>
    <td>+2.7</td>
    <td>+4.0</td>
    <td>+5.0</td>
    <td>+12.0</td>
    <td>+5.5</td>
    <td>+5.2</td>
    <td>-</td>
  </tr>
</table>

Table 7: Results on three hallucination benchmarks. † indicates the factuality hallucination benchmark. ‡ indicates the faithfulness hallucination benchmark. We conduct the experiments based on Alpaca dataset.

D Transferability Study

To verify the transferability of the NOVA method, we conducted experiments on different foundation models using the Alpaca instruction dataset shown in Table 7 and Table 8. We select LLaMA (Touvron et al., 2023) and Qwen-2 (Yang et al., 2024) at the 7B size as the new base models. We aim to gain deeper insights into the applicability of the NOVA method across different models, providing a reference for further research and applications. We find that the NOVA method is also applicable to other models, showing strong transferability and robustness to other models and further research. Compared to other baselines, NOVA significantly reduces hallucinations and keeps a strong ability to follow instructions.

E Design Exploration

The Design of NLI Model We further explore the effects of the NLI model on the final performance of NOVA. We first attempt to analyze the effect of the size of the model on the final results. Specifically, we introduce DeBERTa-base-mnli4, DeBERTa-xlarge-mnli5 and DeBERTA-xxlarge-mnli6. As shown in Table 9, we can find that increasing the size of the NLI model can provide some improvement in the final result, especially when changing the DeBERTa-base-mnli to DeBERTa-large-mnli. However, continuing to increase the model parameters did not have a significant impact on the final performance. Therefore, in order to balance the performance and the inference time of NLI models, we select the DeBERTa-large-mnli to report the final results in our paper. Meanwhile, we further explore whether we use the advanced LLMs (e.g., GPT-4o and GPT-3.5-Turbo) to directly identify the semantic equivalence and get the correct semantic clusters. Specifically, we use the prompt shown in Figure 6 to test the gener-

4https://huggingface.co/microsoft/deberta-base-mnli
5https://huggingface.co/microsoft/deberta-xlarge-mnli
6https://huggingface.co/microsoft/deberta-v2-xxlarge-mnli
<table>
  <tr>
    <th>Model</th>
    <th>MT-Bench</th>
    <th>FollowRAG-Instruction</th>
  </tr>
  <tr>
    <td colspan="3">LLaMA-1</td>
  </tr>
  <tr>
    <td>Vanilla - 100%</td>
    <td>47.8</td>
    <td>37.7</td>
  </tr>
  <tr>
    <td>FLAME-DPO<sup>fast</sup></td>
    <td>40.6</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>SELF-EVAL</td>
    <td>42.2</td>
    <td>38.1</td>
  </tr>
  <tr>
    <td>IFD - 5%</td>
    <td>48.3</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>CaR - 5%</td>
    <td><b>50.1</b></td>
    <td><b>38.2</b></td>
  </tr>
  <tr>
    <td>Nuggets - 5%</td>
    <td>48.6</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td>NOVA - 5%</td>
    <td>49.8</td>
    <td>38.1</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+2.0</td>
    <td>+0.4</td>
  </tr>
  <tr>
    <td>IFD - 10%</td>
    <td>47.9</td>
    <td>38.6</td>
  </tr>
  <tr>
    <td>CaR - 10%</td>
    <td><b>49.5</b></td>
    <td>38.1</td>
  </tr>
  <tr>
    <td>Nuggets - 10%</td>
    <td>48.4</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>NOVA - 10%</td>
    <td>49.3</td>
    <td><b>39.0</b></td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+1.5</td>
    <td>+1.3</td>
  </tr>
  <tr>
    <td>IFD - 15%</td>
    <td>48.5</td>
    <td>38.2</td>
  </tr>
  <tr>
    <td>CaR - 15%</td>
    <td><b>50.3</b></td>
    <td>37.6</td>
  </tr>
  <tr>
    <td>Nuggets - 15%</td>
    <td>49.5</td>
    <td><b>38.6</b></td>
  </tr>
  <tr>
    <td>NOVA - 15%</td>
    <td>48.3</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td>Δ compared to Vanilla - 100%<sup>a</sup></td>
    <td>+0.5</td>
    <td>+0.4</td>
  </tr>
</table>

<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>BioGEN</th>
  </tr>
  <tr>
    <td colspan="3">Alpaca</td>
  </tr>
  <tr>
    <td>DeBERTa-base-mnli</td>
    <td>0.1B</td>
    <td>49.7</td>
  </tr>
  <tr>
    <td>DeBERTa-large-mnli</td>
    <td>0.3B</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>DeBERTa-xlarge-mnli</td>
    <td>0.7B</td>
    <td>50.1</td>
  </tr>
  <tr>
    <td>DeBERTa-xlarge-mnli</td>
    <td>1.3B</td>
    <td><b>50.5</b></td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo-0125*</td>
    <td>unknown</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>GPT-4o-2024-05-13</td>
    <td>unknown</td>
    <td>50.2</td>
  </tr>
  <tr>
    <td colspan="3">Alpaca-GPT4</td>
  </tr>
  <tr>
    <td>DeBERTa-base-mnli</td>
    <td>0.1B</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>DeBERTa-large-mnli</td>
    <td>0.3B</td>
    <td>50.5</td>
  </tr>
  <tr>
    <td>DeBERTa-xlarge-mnli</td>
    <td>0.7B</td>
    <td><b>51.2</b></td>
  </tr>
  <tr>
    <td>DeBERTa-xlarge-mnli</td>
    <td>1.3B</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo-0125*</td>
    <td>unknown</td>
    <td>49.2</td>
  </tr>
  <tr>
    <td>GPT-4o-2024-05-13</td>
    <td>unknown</td>
    <td>50.0</td>
  </tr>
</table>

Table 8: Results on two instruction-following benchmarks based on Alpaca dataset.

<table>
  <tr>
    <th>Model</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td colspan="2">Our Used Reward Model</td>
    <td><b>92.0</b></td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo-0125</td>
    <td></td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>GPT-4o-2024-05-13</td>
    <td></td>
    <td>90.0</td>
  </tr>
</table>

Table 9: FactScore results on BioGEN with different models. We conduct experiments on LLaMA-3-8B and use selected 5% data from different datasets.

Table 10: Accuracy of our used reward model and other advanced LLMs on the constructed test set.

ated responses and the target response by querying the advanced LLMs to identify semantic equivalence. We use the same method as SEI, utilizing the outputs of advanced LLMs to derive semantic clusters and calculate the score of \( F_{res}(r) \). As shown in Table 9, the direct application of results from advanced LLMs proves effective in identifying semantic equivalence. Nevertheless, using NLI models delivers competitive or superior final performance while avoiding API-related costs. Consequently, employing NLI models to identify semantic equivalence is both efficient and effective, substantiating the efficacy of our designed SEI approach.

The Design of Quality Reward Model We also explore the effectiveness of the quality reward model. We introduce UltraFeedback (Cui et al., 2024) and sample 100 instructions and their corresponding responses as the test set (we find that most of the selected data are in English, but some of the selected instruction types are translation tasks, so a few data contain Chinese responses). Specifically, for each instruction, we randomly select 2 responses and determine the ranking between the responses based on their labeled scores of instruction-following, honesty, truthfulness, and helpfulness. Only if all four scores are higher will the response be considered a high-quality response. Meanwhile, we involve two Ph.D. students to conduct the human evaluation to ensure the correctness of the response ranking of each sample. Afterwards, we take the instructions and the responses as inputs to each model, and let the model determine the ranking between the responses and calculate the accuracy of the model’s prediction of the ranking. We compare our used Quality Reward Model with GPT-3.5-Turbo-0125 and GPT-4o-2024-05-13. We use the same prompt for each model as Ge et al. (2024). As shown in Table 10, our reward model achieves better performance, showing the effectiveness of our method. Despite GPT-4o’s strong alignment with human preferences in most general tasks, our reward model trained on the expert-revised preference dataset can perform better, highlighting the subtle gap between expert preferences and advanced GPT-4o preferences.

The Design of Obtaining Sentence Embedding. Alpaca-GPT4 For \( K \) generated responses, we use the internal states of the last token of each response in the last layer as the final sentence embeddings \( E = [e_1, e_2, ..., e_K] \), as it effectively captures the
The Prompt for Identifying the Semantic Equivalence

Please compare the following two sentences and determine whether they are semantically the same. If they are semantically identical, respond with "Identical"; if not, respond with "Different." Consider the meaning, context, and any implicit nuances of the sentences.

Sentence 1: {Sentence 1}
Sentence 2: {Sentence 1}

Provide your judgment below:

Figure 6: The prompt for identifying the semantic equivalence.

<table>
  <tr>
    <th>Model</th>
    <th>BioGEN</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <td><b>NOVA - 5%</b></td>
    <td>50.5</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td>-w. Average Pooling</td>
    <td>49.5</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>-w. The First Layer</td>
    <td>48.9</td>
    <td>63.7</td>
  </tr>
  <tr>
    <td>-w. The Middle Layer</td>
    <td>49.8</td>
    <td>64.4</td>
  </tr>
  <tr>
    <td>-w. The Last Layer (Ours)</td>
    <td>50.5</td>
    <td>64.6</td>
  </tr>
</table>

Table 11: Evaluation results of NOVA that employ various methods for obtaining sentence embedding. We conduct the experiments based on LLaMA-3-8B and the Alpaca-GPT4 dataset. We report the FactScore results on BioGEN.

<table>
  <tr>
    <th>Model</th>
    <th>BioGEN</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <td><b>NOVA - LLaMA-3-8B - 5%</b></td>
    <td>50.3</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>-w/o. Few-shot Demonstrations</td>
    <td>50.1</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td><b>NOVA - LLaMA-1-7B - 5%</b></td>
    <td>43.6</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>-w/o. Few-shot Demonstrations</td>
    <td>41.9</td>
    <td>49.2</td>
  </tr>
</table>

Table 12: The effects of used few-shot demonstrations. We conduct the experiments based on two base models and the Alpaca dataset. We report the FactScore results on BioGEN.

sentence semantics (Azaria and Mitchell, 2023). We further explore the different ways to obtain sentence embedding. Specifically, we first average all the internal states of tokens in the sentence to obtain the sentence embedding (named Average Pooling), which is an intuitive method to get the sentence embedding for decoder-only models. As shown in Table 11, we can find the design of NOVA achieves better performance in both reducing hallucinations and following instructions, showing the effectiveness of our designed SEI. We further explore the internal states from which layer in the LLMs can be used to effectively measure the consistency. Except for the internal states from the last layer, we select both internal states from the first layer and internal states from the middle layer (layer 16 for LLaMA-3-8B), and use the internal states of the last token to represent the sentence embeddings. We can find that using sentence embedding in the shallow layer yields inferior performance compared to using sentence embedding in the deep layers, as the shallow layer may not effectively model the rich semantic information. Overall, extensive experiments show that our design of NOVA is sound and effective.

The Design of Using Few-shot Demonstration.
As detailed in Sec. 3.1, we sample \( K \) responses \([r'_1, ..., r'_K]\) from a base LLM with few-shot demonstrations (Lin et al., 2024a) to ensure the coherence of generated responses. We use the same demonstrations as Lin et al. (2024a). We further conduct experiments to explore the effects of these used demonstrations. We find that using few-shot demonstrations in the process of generating responses for a given instruction allows the base LLMs to better express what they have learned in the pre-training stage. In turn, this will enable ICP and SEI to better estimate the knowledge contained in the instruction data and thus better identify the high-quality instruction data that aligns well with the LLM's learned knowledge to reduce hallucination and improve instruction-following ability. At the same time, we find that this strategy improves more for base models with poor capabilities (e.g., LLaMA-1-7B), which is due to the fact that a poor base LLM may hold relevant knowledge in response to a query, yet occasionally falters in conveying accurate information (Zhang et al., 2024b).

F Human Evaluation

During the human evaluation, the participants follow the principles in Figure 7 to make the decision. For each comparison, three options are given (Ours Wins, Tie, and Vanilla Fine-tuning Wins) and the majority voting determines the final result. We invite three Ph.D. students to compare the responses
The Principles of Human Evaluation

You are asked to evaluate the biographies generated by different models. You should choose the preferred biography according to the following perspectives independently:

1. Factuality: Whether the biography provides relatively more factual statements over the non-factual statements?
2. Helpfulness: Whether the biography provides useful information?
3. Relevance: Whether the statements contained in the biography relevant to the provided people entity?
4. Naturalness: Whether the biography sound natural and fluent?

Finally, please make a decision among 3 opinions, including Win, Tie, and Loss.

Figure 7: The principles of human evaluation.

generated by the models. Before participants begin to make judgments, we describe the principles of our design in detail and ensure that each participant correctly understands the principles. If the final result can not be determined by the majority voting, we will make the discussion among the participants and vote on the result again.

G Case Study for Selected Samples

To evaluate our proposed NOVA qualitatively, we also select some instruction samples from the Alpaca dataset for case studies as shown in Figure 8. Firstly, we can find that simply using \( R_{familiarity} \) in Eq. (10) can effectively identify the simple and straightforward instruction samples that align well with LLM’s knowledge. On the contrary, the sample ranked last according to \( R_{familiarity} \) contains the open-ended instruction and the very subjective target response. Meanwhile, further using \( R_{final} \) in Eq. (10) that considers characteristics beyond just familiarity \( R_{familiarity} \) by introducing the quality reward model further enhances data quality, e.g., the complexity of instructions. We can also observe that the sample ranked last according to \( R_{final} \) also contains the open-ended instruction and the detailed target response, which may introduce familiar knowledge to LLMs and further promote hallucinations.
Case Study for Selected Samples

Sample ranked first according to \( R_{familiarity} \) in Eq. (10):
Instruction: Find a word that is opposite in meaning to the given word: assured.
Target Response: Doubtful.

Sample ranked last according to \( R_{familiarity} \) in Eq. (10):
Instruction: Could you pick a set of five songs which will fit our theme: a summer night’s party?
Target Response:
1. Billie Eilish – “Bad Guy”
2. Ed Sheeran – “Shape of You”
3. Maroon 5 – “Summer Nights”
4. DNCE – “Cake by the Ocean”
5. Drake – “In My Feelings”.

Sample ranked first according to \( R_{final} \) in Eq. (10):
Instruction: Given a dictionary, create a JSON representation. dict = {"name": "John", "age": 18, "is_adult": True}
Target Response:
{
    "name": "John",
    "age": 18,
    "is_adult": true
}

Sample ranked last according to \( R_{final} \) in Eq. (10):
Instruction: Describe how to impress a customer.
Target Response: Listen to the customer carefully, acknowledge their needs and desires, make sure to follow up on commitments, and strive to exceed expectations.

Figure 8: Case study for selected samples.

Reviewer paper summaries: Due to the unfamiliarity of the instruction samples, a PLM can hallucinate the answers based on the instruct finetuning data. Therefore, this paper proposed one method to filter out some samples that the PLM is not familiar with. To achieve the goal, the framework mainly includes four steps: sampling answers from a PLM, internal consistency probing, semantic equivalence identification and data quality estimation. The results show the effectiveness of the proposed method.
Summary of strengths: 1. The result supports the main claim because it balanced the hallucination and instruction following.

2. The idea is reasonable because it follows the observation that unfamiliar samples can lead to hallucination.

3. The demonstration is clear and easy to follow.
Summary of weaknesses: 1. The motivation of using Gaussian distribution to model the sentence embedding seems weak. Modeling the embedding uncertainty by Gaussian imposed a strong symmetric assumption. Gaussian indeed can be a common assumption. But more exploration on this will make the method stronger.

2. It is unclear how to determine the cluster beforehand. If the cluster number is unknow, are there any fast way to estimate it?

3. The analysis of the complexity of the dataset selection is missing. It is important to know if the dataset selection is more complex than using all data and comparing with other baselines.

4. It is still unclear how the model will behave if an unfamiliar question is given. Will the model generate something very short to reduce the risk of hallucination?
Comments, suggestions and typos: N/A
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper proposes a method to align LLM for better instruction following and ease hallucination via data filteration. The authors introduce two specific mechanisms to let the the model better choose data samples for LLM training so to improve the model confidence and better contain correct knowledge. The experiments show the proposed NOVA can indeed achieve the expected results.
Summary of reasons to publish: Reviewers all agree that this paper has its clear strengths, such as the novelty of this method, the insightful approach, and also the strong experimental improvements.