Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
Authors: Yuzhe Gu
Wenwei Zhang
Chengqi Lyu
Dahua Lin
Kai Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Hallucination Mitigation
Large Language Model
Fine-grained Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: This paper proposes a fine-grained hallucination mitigation framework based on Direct Preference Optimization (DPO), incorporating sentence-level factuality as reward signals.

Abstract: Large language models (LLMs) exhibit hallucinations (i.e., unfaithful or nonsensical information) when serving as AI assistants in various domains. Since hallucinations always come with truthful content in the LLM responses, previous factuality alignment methods that conduct response-level preference learning inevitably introduced noises during training. Therefore, this paper proposes a fine-grained factuality alignment method based on Direct Preference Optimization (DPO), called Mask-DPO.  Incorporating sentence-level factuality as mask signals, Mask-DPO only learns from factually correct sentences in the preferred samples and prevents the penalty on factual contents in the not preferred samples, which resolves the ambiguity in the preference learning. Extensive experimental results demonstrate that Mask-DPO can significantly improve the factuality of LLMs responses to questions from both in-domain and out-of-domain datasets, although these questions and their corresponding topics are unseen during training. Only trained on the ANAH train set, the score of Llama3.1-8B-Instruct on the ANAH test set is improved from 49.19% to 77.53%, even surpassing the score of Llama3.1-70B-Instruct (53.44%), while its FactScore on the out-of-domain Biography dataset is also improved from 30.29% to 39.39%. We further study the generalization property of Mask-DPO using different training sample scaling strategies and find that scaling the number of topics in the dataset is more effective than the number of questions. We provide a hypothesis of what factual alignment is doing with LLMs, on the implication of this phenomenon, and conduct proof-of-concept experiments to verify it. We hope the method and the findings pave the way for future research on scaling factuality alignment.

Published as a conference paper at ICLR 2025
MASK-DPO: GENERALIZABLE FINE-GRAINED FACTUALITY ALIGNMENT OF LLMS
Yuzhe Gu!.2 Wenwei Zhang
Chengyi Lyu²
Dahua Lin Kai Chen
1 Shanghai Jiao Tong University Shanghai Al Laboratory 3 MMLab, The Chinese University of Hong Kong
(guyuzhe, zhangwenwei, vychengqi, chenkai (otpakiiiiplllbooo...oo org.co
ABSTRACT
Large language models (LLMs) exhibit hallucinations (i.e., unfaithful sensical information) when serving as AI assistants in various domains. Since hallucinations always come with truthful content in the LLM responses, previ- ous factuality alignment methods that conduct response-level preference learning inevitably introduced noises during training. Therefore, this paper proposes fine-grained factuality aligmment method based on Direct Preference Optimiza- tion (DPO), called Mask-DPO. Incorporating sentence-level factuality mask signals, Mask-DPO only learns from factually correct sentences in the preferred samples and prevents the penalty on factual contents in the not preferred samples, which resolves the ambiguity in the preference learning. Extensive experimental results demonstrate that Mask-DPO can significantly improve the factuality of LLMs responses to questions from both in-domain and out-of-domain datasets, although these questions and their corresponding topics are unseen during training. Only trained the ANAH train set, the score of Llama3.
on the ANAH test set improved from 49.19% to 77.53% even surpassing the score of Llama3. 1--00B-Insttttt (53.44%), while its FactScore on the out-of-domain Biography dataset is also improved from 30.29% to 39.39%. We further study the generalization property of Mask-DPO using different training sample scaling strategies and find that scaling the number of topics in the dataset is more effective than the number of questions. We provide hypothesis of what factual alignment is doing with LLMs, on the implication of this phenomenon, and conduct proof- of-concept experiments to verify We hope the method and the findings pave the way for future research on scaling factuality alignment. Code is available at https::/gittiini com cow/oper: oper: Ccomaass/ANe ANAH
INTRODUCTION
Large Language Models (LLMs) have demonstrated impressive performance across various tasks (Ka- malloo et al. 2023 Sun et al. 2023 Chen et al., 2024). Even though, LLMs still face concerning issue: hallucination, in which they generate plausible-sounding yet inaccurate nonsensical infor- mation (Ji et al. 2022 Bang et al. 2023) in response to user queries, particularly those demanding extensive knowledge. The hallucination phenomenon has two main characteristics (Ji al. 2024 Mishra et al. 2024) that significantly impede the real-world application of LLMss: 1) the unfaithful or nonsensical information always exists with truthful contents the LLM responses, posing a challenge to accurately and effectively detect and mitigate them; 2) LLMs exhibit hallucinations across various domains and tasks, but it difficult and impractical to thoroughly mitigate them by exhausting all real-world knowledge. The prevalence of these two issues underscores the need to develop effective and generalizable methods for fine-grained factuality alignment to reduce hallucinations Large Language Models (L.LMs).
Existing methods (Tian al., 2023 Lin et al. 2024 Zhang et al., 2024b) Chen Li, 2024 mainly
apply preference learning- based method, especially Direct Preference Optimization (DPO) (Rafailov et al. 2024), to align the internal knowledge of LLMs to the facts. These approaches utilize the
Corresponding author

---

Published as a conference paper at ICLR 2025
incorrect reward signal
correct sentence
DPO
correct reward signal
incorrect sentence
Mask-DPO
8
B
Enoourage
Sentl
Y Senti
Penalize
Encourage
Sentl A
Encouzage
Sant2 v
X Sent2
Penalize
Encourage + Sent2 -
X Sant2
Penalize
Encourage
Sent3 X
X Sent3
Penalize
Smptt X
X Sent3
Penalize
Preference Data
Fine-grained Preference Data
(a)
(b)
Figure 1: Comparison between DPO and Mask-DPO. Vanilla DPO (a) inadvertently encourages and penalizes all the content in the preferred and non-preferred samples, respectively, regardless of their correctness. Instead, Mask-DPO (b) incorporates sentence-level facticity into the mask signal, preventing incorrect reward signal, which resolves ambiguity in preference learning.
response-level factuality for constructing pairwise preference data for preference optimization, which aims to maximize the probability of preferred samples with higher factuality while minimizing the probability of non-preferred samples with lower factuality. However, since both factually correct and incorrect sentences are often mixed within a single response (Ji et al. 20224 Mishra et al. 2024). vanilla DPO inadvertently and inevitably encourages incorrect information in the preferred samples and penalizes truthful ones in the non-pree con-prffrree samples (Figure Tigure[a)). Such ambiguity in the learning completely wrong correct, which is often the case,
exists as long as the training samples ultimately reducing the effectiveness of factuality alignment.
To address the issue, this paper proposes a fine-grained factuality alignment framework, named Mask-DPO. As shown in Figure unlike vanilla DPO, Mask-DPO leverages sentence-level factuality information as mask signal to resolve the ambiguity in preference learning. Specifically, in the preference data construction stage, Mask-DPO uses fine-grained hallucination annotator (e.g., ANAH-v2 (Gu et al. 2024) to determine the factual correctness of each sentence, which will be used to guide the preference learning later. The responses that contain more and fewer factually correct sentences will chosen as preferred and non- non-preferred samples, respectively, in preference pair for learning. During the preference learning stage, the sentences that cause ambiguities in training, i.e. incorrect sentences the preferred samples and correct sentences in the not preferred samples, would be ignored, guided by the sentence-leve mask annotated in the previous stage. Thus, Mask-DPO can avoid encouraging incorrect sentences when maximizing the probability of the preferred sample, and avoid penalizing correct sentences when minimizing the probability of the non-preferred sample.
Through extensive experiments, we demonstrate that Mask-DPO significantly enhances the factuality of LLMs and exhibits more effectiveness than DPO. Mask-DPO boosts the score of Llama3.1 1- 8B-Instruct on the ANAH test set from 49. 19% to 77.53%, surpassing the Llama3. 1ormaa.1-70B-nae -OOt-Inrrttt (53.44%) and DPO (68.44 448,, although the questions and topics in the test set are unseen during the aligmment. Moreover, when testing the same model on the oft-of-domain Biography dataset, 1-mma3.--B-IIssttee aligned by Mask-DPO also exhibits excellent performance, whose FactScore improved from 30.29% to 39.39% reaching a level close to Llama3, 1lama3.1-700 -n-trrutttt (40.47%).
We further study the scaling and generalization property of Mask-DPO, by scaling the training data through two dimensions: the number of different topics and the diversity of questions within the same range of topics. The experimental results show that scaling the number of topics more effective for improving the factuality and generalization of models, in comparison to increasing the diversity of questions under the same topics, under the constraints of using the same amount of preference pairs. On the implication of this phenomenon, knowledge graph during training with the language modeling objective, where different topics and their corresponding information be regarded graph nodes, and the affinity between two nodes positively determines the probability of blurring their corresponding knowledge when the LLM responds to the questions related either one of the topics.
hypothesize that each LLM Iearns model-specific
Under this hypothesis. factual alignment on a topic essentially adjusts the affinity between the topic and its nearby nodes, which will also help improve the factuality when the model answers questions
2

---

Published as a conference paper at ICLR 2025
Stepl: Data Generation
Lengrexdew town Jocated Sampden Countyy Massecuuatts. Approxinataaly
foringtield.
Response
Describe the geographic location of Longratdoo ia relation to apringfield and Hartford.
Candidate Respondes
Sampling Resporses from Policy Model
Sentli Longroalioe lo a sown located in Sampdlnn Councy, Miseadluueetoo Sent2: 2z sa about: 15 milea nezta of Barsford.
Question
Sent3: 22 la picoated 5 ailes Bouth of speingfield.
Response
Step2: Fine-grained Preference Data Construction
Step3: Fine-grained Preference Learning
Responde
Sentence Level Factual Annotation
incoceact
Preformed y
Non Prefermed y
Scoring Ranking
Masking Some Anechronistic Sentences
Precessing Signols
Response
Senti Sent2 Sect
Encouuage
Senti
Annotator
Senti o
Soati Y
Preference Pair
0.67
Senti
San
Penalice
Bent2 of
Seat2 X
K 0.33
X Sent
Penal lie
X
X
Preference Learning
Maximum
Preferred y
Hampdea Massocusetts.
iuperosiimacly 28 mika North Hartthed And 7 mikes Seath Springfied....
Reference
Non-Preferred
More Truthful LLM
The overview of Mask-DPO. First, we sample K candidate responses for each question from the policy model. Then, we use fine-grained hallucination annotator to perform a sentence- level factuality annotation on each response. We use the proportion of correct sentences out of the total number of sentences as the factuality score. We select the responses with the highest and lowest scores as preferred and non non-preferred samples, respectively. Finally, we perform fine-get gra-gra factuality alignment the policy model using such fine-grained preference data, where the reward signals to the sentences, i.e., incorrect sentences in the preferred samples and correct sentences the non-preferred samples, would be ignored.
Figure 2:
of the nearby topics, even if they are not seen in the alignment, We conduct two proof-of-concee experiments verify our hypothesis. The first one ablates different topic sampling strategies and shows that sampling the nearest topics to those in the test set based on the model-specific clustering can most effectively improve the factuality score on the test set. The comparison between the factuality of best-of-N responses of the model before and after factuality alignment further verifies that the internal knowledge of LLMs is indeed more aligned to the facts after factuality alignment, which has not been observed in previous experiments of preference learning other domains such as reasoning (Havrilla et al. 2024). We wish Mask-DPO and our findings could pave the way for future research on scaling factuality alignment of LLMs.
MASK-DPO
is built on a meference-basee reinforcement learning framework ($ 2.1). While previous methods optimize preferences at the response level, our approach employs dense supervision the sentence level and performs a fine-grained preference learning($ 2.2). as shown Fig.
2.1 PRELIMINARIES
Reinforcement Learning from Human Feedback (RLHF) is a powerful alignment method for fine- tuning language models to enhance their robustness, factuality, and safety (Ouyang et al. 2022). In practice, given a prompt æ and a corresponding LLM response y, RLHF aims to maximize the following objective:
Te (vu (z)
max E p...D.o... 16
r(x, y) - 3 log
Tee (yn
where Dp represents the prompts dataset, To is the policy model to be optimized, Trey is the reference model used for regularizing Te with Kullback-Lan Leibler divergence and is constant to control the degree of regularization. The reward model r reflects human preferences, which takes a prompt and the corresponding response input and outputs scalar value.
3

---

Published as a conference paper at ICLR 2025
However, training reward model and integrating into the overall pipeline can be quite com- plex (Zheng et al. 2023). To avoid this, Rafailov et al. (2024) proposed Direct Preference Optimiza- tion (DPO), which directly optimizes the policy model using preference pairs. Given a prompt a preference pair that contains Yu and sampled from the policy model тo. where yhe has higher quality than y/- DPO aims to maximize the probability of the preferred sample you while minimizing the probability of the less desirable sample yt- The optimization objective is formulated as:
Tee (b) B log
Ta Teer(3/ (z)
LTPO Teet) - (w. log
log
(2)
Tree(yel x))
where D is the pair-wise preference dataset and is the sigmoid function.
Preference learning-basee alignment has been proven effective for factuality tuning mitigating the hallucination of LLMs (Tian et al.) 2023) Lin et al.) 2024] [hhng et al. (2024b). However, the
optimization objective of Eq. may not be entirely suitable for knowledge- baseedde-baed tasks. Taking the The log of its
example, we assume it has N sentences, i.e. generative probabilities in Eq. can be expanded as:
preferred sample Yu
To (Yu (x) Tee V/o (z)
log I1 i=
$1
18 81-1-1)
log
log (S:( 81wi-1)
i==1 Tre 81mi-1)
there is sentence in twr that contains incorrect facts, the generation probability of 8; would be maximized when maximizing the generation probability of yee Similarly, the generation probability of the correct sentence in y would be minimized when minimizing the y generation probability. This affects the effectiveness of factuality alignment.
In such
2.2 FINE-GRAINED FACTUALITY ALIGNMENT
Fine-grained Preference Data Construction. We perform fine-grained factuality annotation using sentence-level hallucination annotator (or reward model). Given the prompt æ and its corresponding response y. which consist of N sentences { the reward model A annotate each sentence and give the fine-grained feedback {a, N which can be formulated as:
(a: J {A(s;,
x) ())
A(y,
Here, we let a; 0 represent that sentence has hallucination, and a; represents that it has the hallucination.
Thanks to the fine-grained factuality annotation, we can now construct the preference dataset for alignment. First, for each prompt æ, we perform top-k sampling to select candidate responses Ji=n from the policy model. For each candidate, annotate it according to Eq. and obtain (z. y", 8" a'K Ji=o Next, we compute the factuality score for each candidate, which we define as the fraction of correct facts: (1=)() where N is the number of sentences in candidate y². Then, we construct a preference dataset based on factuality scores, where we assign y with higher scores to Yo and the lower ones to yt- Finally, we filter the data to filter out preference pairs where you and y the same, Vou does not contain correct facts, and z does not contain incorrect facts.
y [y' K
Fine-grained Preference Learning. Using the constructed fine-grained preference dataset D {z, yue Sw, aws Vt, $1, a.), Mask-DPO forces the model to only learn from the correct facts in the preferred examples You and the incorrect contents in the un-preferred examples yt- Based on Eq. designed masking scheme applied to the DPO algorithm as:
To(s; xr 81
Mw( Swi aw) = Σ I(a; = 0) log
Tiet 1., S1w1-1)
To(s} x Sjnj-1)
Mi(2, Swi aw) = EI(a, = 1)
neeh (3 x $1n5-1)
where Ma is the masked KL divergence for Yo while Mi for yı- Combining Eq. 2 and Eq. formulate the optimization objective of fine- tne-grained factuality tuning:
we
LM DPO ( ne))) - au (1,0)) (log (BMw Sw: aw) (M,(
(6)
4

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results for the open-source models, FactTune, and our Mask-DPO. Here, ANAH and Biography represent the in-domain and out-of-domain test sets, respectively. ANAH- v2 and FactScore represent the corresponding evaluation strategies. For each evaluation strategy. we report the number of correct facts (# Correct.), the number of inaccurate facts (# Inc.), and the factuality score (% Score), i.e. the proportion of correct facts out of the total number of facts. Bold and underlined represent the best and second best performance, respectively.
ANAH (in-domain)
Biography (out-of-domain)
Model
ANAHI-v2 Evaluator
Faciscore Evaluator
Pactscore Evaluator
Cor.
Inc.
% Score
# Cor.
# Inc.
% Score
# Cor.
# Inc.
% Score
Qwen2-7B
450
872
34,03
5.19
30.60
15.57
14.46
39.27
27.28
Qwen2-72B
538
690
43.81
6.45
27.97
20.76
20.13
40.09
34.06
Gemma2.9B
635
837
43.13
4.51
25.92
18.29
15.53
29.00
29.52
Gemma2-27B
889
1110
44.47
7,81
35,65
20.14
19.27
36.59
29.99
Yi1.5-6B
440
1143
27.79
6.01
44.48
11.92
10.84
59.65
15.81
Yi1.5-9B
483
1136
29.83
5.06
39.79
10.42
10.63
62.67
15.33
Yi1.5-34B
535
911
36,99
36.96
13.27
17.01
52.76
25.49
L-8
461
476
49.19
5,95
21.29
19.43
16.83
31.57
30.29
Llama3.1-708
520
453
53.44
6.81
22.17
21.92
23.58
31.59
40.47
FactTune
657
499
56.83
7.45
23.08
22.67
15.93
23.97
37.97
Ours
547.6
161.4
77.53
5.90
18.59
25.56
12,16
15.24
39.39
3
EXPERIMENT
3.1
EXPERIMENT SETUP
Implementation. our experimental framework, we adopt the 1aamma3.--B--Isttuuue (Dubey et al. 2024) the base model and the ANAH-v2 (Gu et al. 2024) as the fine genennra reward model to annotate factuality. All the experiments are conducted using the same setting for a fair comparison. Further implementation details can be found in Appendix
Dataset. For the in-domain data, we use a subset of the ANAH-v2 (Gu et al. 2024) data as the test set, containing 177 questions, and another 8046 questions the training set. ANAH organized along two dimensions: topics and questions, and the training set has no overlap with the test set in both dimensions. For the out-of-domain data, we use subset of Biography (Min et al. 2023) as the test set, which has 183 questions about biography generation and has no overlap with our training set.
Evaluation. To evaluate each generated response, we use counts of correct and incorrect facts computed by ANAH-v2 (Gu et al. 2024) and FactScore (Min et al., 2023) as the evaluation metrics. ANAH-v2, which was trained specifically on our in-domain data for the hallucination annotation task, would annotate each sentence in the response with hallucinations, and we used the ratio of non-hallucinated sentences the total number of sentences as the final score. Since ANAH-v2 has been used in preference data construction, to avoid reward hacking, we also use FactScore for evaluation, which calculates the number of correct facts and reports its proportion. In addition, considering that ANAH-v2 has not yet been trained in the task of generating biographies, and to ensure the reliability of the results, we only use FactScore as an evaluation strategy when evaluating out-of-domain data. For stability, we default reporting the mean value after five replications.
Baseline. We construct the baseline in two ways. First, we compare Mask-DPO with many open- source models, including Qwen2 (7B & 72B) (Yang et al. (024a)), Gemma2 (9B & 27B) (Team et al.) 2024), Yi1.5 (6B & 9B & 34B) (Young et al. 2024), and Llama3. (8B & 70B) (Dubey et al.) 2024). Note that all open-source models used here are post-trained versions and we only report results from single experiment for them. In addition, we also compare FactTune (Tian et al.) (202). hallucination mitigation method that also based on DPO. We apply the same base model.
3.2 OVERALL RESULTS
Evaluation on In-Domain Data. The first six columns of Tabel illlstrate the performance of the open source models, FactTune, and our Mask-DPO DPO on the in-domain data. Under the ANAH-v2 evaluation strategy, Mask-DPO achieves the highest factuality score (77.53%). surpassing the best
5

---

Published as a conference paper at ICLR 2025
Table Ablation study for the masking scheme applied to the DPO. Here, "w/ mask" means that the factual alignment algorithm is DPO with mask, which is the default setting of Mask-DPO. "w/o mask" means that the algorithm is the vanilla DPO.
Training Setting
ANAH-v2
FactScore
# Correct
# Incorrect
% Score 1
# Correct
# Incorrect
% Score 1
w/o mask
446.60
206.00
68 44
5.71
18.98
23.43
w/ mask
547.60
161.40
77.53
5.90
18.59
25,56
Table 3: Ablation study for the sampling strategy in preference dataset construction. Here, "Llama" means that the data used to construct preference pairs is sampled from the policy model (Llama3.1 I-st-rruuct,, which is the default setting of Mask-DPO. "Internl.M" means that the data is sampled from MntemLLM--BB-hha-- policy model, but the sampling strategies for preferred and dispreferred samples are different. When generating preferred samples, the reference document corresponding to the question is added to the prompt, but not to the dispreferred ones.
And "Llama+Doc" means that the data is sampled from the
Sampling Setting
ANAH-v2
FactScore
# Correct
Incorrect
% Score f
Correct
Incorrect
Score t
InternI
419.20
296.80
59.43
5.19
16.59
21.33
Llama+Doc
877.00
1347.00
39.43
7.98
33.07
16.16
Llama 547.60
161.40
77.53
5.00
18.59
25.56
open-source model Ilama3. 1-70--Instttet (53.44%) and factuality alignment method FactTune (56.83%). addition, under the FactScore evaluation strategy, the results also show the same trend, where Mask-DPO reaches the highest factuality score (25, (25,56%), surpassing the best open-source model Llama3. Llama3.1-70B-- 1-Ot--Intttttt (21.92% and factuality alignment method FactTune (22.67%). These results also show that our reward model has not been overly and seriously hacked.
Tabel also provides the factuality levels of existing open-source models, among which the Llama3. series achieves the highest factuality level, while the Yil.5 series has the lowest level. Moreover, as the scale of model parameters increases, the factuality level of the model also increases, which consistent with the observations Gu et al. (224)).
The last three columns of Tabel[illlsstie the performance of the open source model, FaciTune, and our Mask-DPO on the out-of-domain data. Without using the corresponding training set, Mask-DPO still exhibits excellent results, improving the factuality the Biography dataset from 30.29% 39.39%, surpassing the factuality alignment method FactTune (37.97% and reaching a level close to the best open-source source model 1lama3.1-70B ----u--nntttutt Instruct (40.47%).
Evaluation on Out-of-Domain Data.
score of Llama3. 1-8B- Instructt
Moreover, the trends the factuality levels between different series of open-source models and between different parameter amounts in the same series consistent with those in Table further illustrating the reliability of our results.
3.3 ABLATION STUDY
Impact of Mask Scheme. To verify the effectiveness of the masking scheme (introduced in $2.2). we compare the performance the models trained with and without the masking scheme. As shown in Tabel 2 the scheme with the mask is significantly better than the scheme without the mask in both evaluation strategies, which demonstrates the effectiveness of MPsk-DPO.
It is worth noting that Fact'Tune (the second to last row in Table [ also uses the vanilla DPO for factuality alignment, but its final performance not as good as the vanilla DPO this experiment (the first row in T2. The main difference between them is that FactTune uses FactScore to construct preference data, while the vanilla DPO in this experiment uses ANAH-v2. Because FactScore and ANAH-v2 are also our two evaluation models, this result is shown in both evaluation strategies, which rule out the possibility reward hackers. Therefore, we believe that the reward model used to construct preference data will have obvious impact on the effectiveness of factuality alignment.

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation for the efficiency and the generalization effects of different training sample scaling strategies. Here, "Topic" means scaling from the number of different topics, and "Question" means scaling from the number of different questions under the same topic. "Topic" denotes the number of topics used for training, and Question" denotes the number of questions under the same topic used for training. We also report changes in scores (in parentheses) when scaling the data.
Dimension
# Topic
Question
Scale
# Correct
# Incorrect
% Score ?
894
1/3
501.80
270.20
65.04
Topic
1788
2/3
518.00
206.20
71.56 (+6.52)
2682
1
547.60
161.40
77.53 (+5.97)
2682
1/3
486.00
218.00
69.03
Question
2682
2/3
444.00
164.00
73.03 (+4.000
2682
3
1
547.60
161.40
77.53 (+4.50)
Impact of Sampling Strategy. To assess the impact of the data sampling strategy (introduced in
$ $2.22. we compare the performance of the models trained with different data sampling strategies in
Tabel 3 In our default setting, we sample the data from the policy model (Llama) to construct the factual preference dataset. We also test two other different sampling strategies, sampling from the non-policy model (InternLM) and sampling from the policy model but using different contexts for preferred and non-preferred samples (Llama + Doc).
As shown in Tabel 3 the performance of sampling from the non-policy model is far less effective than sampling from the policy model, even though it can improve on the base model. Notably, for settings that sample from the policy model but use different contexts for preferred and non-preferred samples, the scores after factuality alignment even drop from the base model. Intuitively, the quality of preference pairs constructed in this setting should be higher due to the presence of corresponding reference documents in the prompt while generating the preferred samples. This may be the case because the prompts used to generate the preferred and non-preferred samples are not the same, resulting in negative impact on the factuality alignment although the quality of the preference pairs higher. This phenomenon is also observed by Lin et al. (2024).
4 ANALYSIS OF DATA SCALING
We further analyze the efficiency and the generalization effects of different training sample scaling strategies of Mask-DPO 4.1. To explain the observed phenomenon of generalization, we provide hypothesis that discusses how factuality alignment implicitly impacts the internal knowledge structure in LLMs ($ 4.2). and then briefly design a series of proof-of-concept experiments to verify it. ($ 4.3)
4.1 ANALYSIS
In our training set, data are organized along two dimensions: topics and questions, where each question be categorized into topic, which is usually the entities the question asks about. Both topics and questions in the training set have no overlap with that of the test set. We analyze the impact of scaling the training data through these two dimensions, including the number of different topics and the diversity of questions under the same topic, respectively.
In the default setting of Mask-DPO, we use 2682 topics for factuality alignment, where each topic contains 3 questions. In the scaling of the number of topics, we use one- one-third, two-thirds, and the full number of topics for alignment. In the question dimension, we use the same scaling strategy, i.e,, only use one, two, or three questions per topic for alignment. Since we have demonstrated the consistency of ANAH-v2 and FactScore terms of results in the previous section ($ for resource considerations, only use ANAH-v2 for the evaluation afterward.
As shown in Table 4 we observe that scaling the number of topics more effective than increasing the diversity of questions. When scaling the same amount of data, scaling the topic leads to larger increases (6.53 and 5.97) compared to scaling the question (4.00 and 4.50). Moreover, the scores for the "scaling number of questions" setting consistently higher than those for the "scaling number of topics" " with the same amount of training data. Since the former consistently covers the full amount of topics, it suggests that the diversity of topics is more important than the diversity of questions.
7

---

Published as a conference paper at ICLR 2025
4.2 HYPOTHESIS
The experimental results in previous sections indicate that after factuality alignment with questions from a group of topics, the LLM will become more factually correct when answering questions from another group of topics. On the implication of this phenomenon, we hypothesize that the factuality alignment essentially adjusts a model-specific graph consisting of topics as nodes represented by the LLM,, where preference learning on some of the topics (nodes in the graph) will be propagated to other nodes through the graph.
Specifically, consider a graph G2 (V, E), where V is the set of nodes, each representing distinct topic and their corresponding knowledge, and S V V is the set of edges, representing relationships between topics. The edges are defined by the affinity between these topics. which are learned specifically for each model through the language modeling objective in training (especially pre- vre-training). Thus, the hallucination can be interpreted as the blurring of knowledge between node and other nodes. For any two topics u, v E V, given the user query x, the hallucination phenomenon can be formulated as:
h(,, v u) 8; mo (: B. 81-i-1),, 81 U. 84 E u.
where 81~i-1 represents the first sentences in the reply of L.M,, related topic u, and 8; represents the jth sentence, which erroneously shifts to information of topic u due to the boundary blurring between topics and v.
Let an embedding function Φ + E maps each topic v E V to high-dimensional vector Φ(v) in the embedding space Id. Due to the duality of the graph structure, the relationship between and u can be quantified by distance metric D, which we hypothesize can be approximated by the probability of its hallucination phenomenon:
((o(u),(v)) & P(h(a, a - u)) s P(h(z, v + u)) = To (8s 11, 81 E U, 8 E
(8)
We hypothesize that more related topics will have smaller distances in the embedding space repre- sented by LLMs, and thus more likely to exhibit hallucination. This also explains that often behaves plausibly and is hard to detect (Ji et al. 2022) due to the proximity of and c.
With this formulation, factuality alignment using Eq. 2 and Eq. 3 on topic decreasing the probability of h(z, v - u). Due to the duality of the affinity between nodes (Eq. 8). nearby topics like u are implicitly adjusted together, which explains the generalization phenomenon. Moreover, Mask-DPO introduces a more accurate adjustment on each topic using Eq. a|d Eq. 6 which makes the factuality alignment more effective.
be considered as
4.3 PROOF OF CONCEPT
If the model-specific graph does exist, and interpretation of factuality alignment the graph is correct, then there will be two corollaries: 1) when factuality alignment is performed on a topic, topics closer to it will be more affected, as evidenced by the factuality score of that topic. factuality alignment does not simply change the generation probability distribution or the style of certain content but has modified the internal knowledge structure of LLMs, which can be observed through best-of-N sampling.
Proof of Corollaries (1). To assess the impact between topies, we design an experiment based on topic clustering. We use different embedding strategies to get the high-dimensional vector of each topic, including embeddings using the policy model itself, embeddings from other models, and embeddings from specialized word embedding models. Then, we cluster the topics in the training set using the topic vectors in the test set as cluster centers, and split the training set into two groups, namely the far group and the near group, according to the distance from the corresponding topic vector to the cluster center. We also use a random strategy to select two groups as baselines. Finally, use the far group and the near group for factual alignment under the same settings.
We conduct experiments using Llama3.1-88 1-1---8B--niie Instructttt (Dubey et al. 2024) and I2temLLM--7B-SE Chat (Cai et al. 2024) as the base (policy) models and use ANAH-v2 for evaluation.
8

---

Published as a conference paper at ICLR 2025
Table 5: Evaluation for the effects of different clustering strategies for preference data con- struction. Here, "IntermLM2-7B", "Llama3. 1-8B" and "OpenAI" denotes Alamm3.1-8B 8B-Instruct and Bext-Embeddiing---aee selecting samples. "Distance" represents the distance between the training set and the cluster center. "Diff" represents the difference between the score near distance settings and the one far.
respectively. And "Random" means randomly
Base Model
Embedding Model
Distance Correct
# Incorrect
% Score t Diff
far 493.10
267.60
64.99
M22-r
2.43
456.40
222.30
67.42
far 454.80
252.20
64.32
Llama3.1-3 1-8B
0.65
487.00
263.00
64.97
Internl M2-778
far 487.64
256.09
65.59
OpenAl
1.25
near 489.50
242.70
66.84
453.60
247.40
64.78
Random
0.77
521.70
275.90
65.55
far 455.50
206.50
68.97
M2277
0.22
near 551.75
280.75
69.19
Jlama3.1-80 far 568.00
265.75
68.99 68.94
4.49
500.00
182.33
73.43
Llama3.1-8B
far 761.25
317.00
71.32
OpenAI
0.85
near 616.60
253.60
72.17
566.00
219.00
70.60
Random
0.56
454.00
182.33
71.11
Table Evaluation for the performance under the setting of best-of-N. Here, "Baseline" repre- sents the base model Llama3.
and "Mask-DPO" represents the aligned model.
Model
Best of 1
Best of 16
Best of 32
Best of 64
Best of 128
Best of 256
Baseline
47.79
73.38
76.30
76.93
79.42 93.47
80.98
Mask-DPO
70.67
89.80
90.53
92.81
94.09
As shown in Table regardless of the embedding strategy used, the factual score of alignment using the near group higher than that using the far group. This indicates when factuality alignment performed node, it affects other nodes, and the effect becomes smaller the distance between nodes increases. Furthermore, when the base model and the embedding model are the same, the score difference between the far and near groups is the largest, i.e., 2.43 for InternLM2-7B and 4.49 for Llama3.1 1-BB, and the near group obtains the best results (67.2% for InterL and 73.43% for Llama3.1-8B3 while the score differences for the other strategies are close those for the random strategy and the near group chose by other strategies cannot obtain the highest results. This suggests that the graph-like knowledge structure is model-specific.
Proof of Corollaries (2), We further test the best-of-N performance of the Llama3. 1-8B- Instruct (Dubey al., 2024 before and after training, using ANAH-v2 for evaluation. Specifically, we perform top-k (k = 40) sampling in inference time, and chose the most factual response from N candidates as the final response for each question.
As shown in Table 6 the best-of-N scores for both settings rise in parallel as the number of samples increases. However, the difference in scores before and after factuality alignment always present and does not gradually converge as the number of samples increases. This is inconsistent with what Havrilla et al. (2024) observed the math reasoning task, where the scores before and after training would converge the number of samples increased, which means that the training only changes the probability distribution that generates the correct answer. Such a phenomenon proves our corollaries that factuality alignment has an essential effect on the knowledge structure within LLMs and does not simply change the generation probability distribution of certain content.
'The results for best-of-1 do not match those in Table because the inference setting here is top-k=40, whereas in Table iit top-k=1.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Reinforcement Learning from Human Feedback. In order to unlock the capabilities of LLMs and align them with human preferences, a bunch of work about alignment has been proposed. Reinforcement Learning from Human Feedback (RLHF) (Quyang et al.] 2022 Bai et al.) 2022] Rafailov et al., 2024; Yuan et al. | 2024) the typical alignment method, a representative work is Proximal Policy Optimization (PPO) (Schulman et al. 2017). PPO uses paired data based on human preferences to train a reward model and then uses the signals provided by that reward model to guide the optimization of the policy model, which is complex. For simplicity, Direct Preference Optimization (DPO) (Rafailov et al.] (002)) utilizes pairwise preference data directly for model optimization. Since using humans for preference data construction is costly (Min et al.] (023). Reinforcement Learning from Al Feedback (RLAIF) (Lee et al. (003) is proposed, which constructs preference data using AT (e-g- advanced LLMss, specialized evaluator models, and agents system) feedback. Our approach can be regarded as a fine-grained RLAIF method for hallucination mitigation, which uses feedback from the hallucination annotator as the reward signal and we improve DPO for more effective factuality alignment.
Hallucination Mitigation. Considering the harm of hallucinationss, researchers have explored various techniques for enhancing the factuality of LLMs. To evaluate the factuality of LLMs, researchers have developed various benchmarks (Ji et al, 2024, Wei et al., 2024, Min et al., 2023) and hallucination
detection methods (Gu et al. 2024; Min et al. 2023). However, these methods only detect issues
without providing solutions. For mitigation, multi-task learning (Garg et al. 2019 Weng ct al. 2020).
model editing (Daheim et al. 2023: Ji et al. 22023), and factuality alignment (Wu et al. [023] Tian
ct al. 2023 Lin et al.] 2024; Zhang et al.] 2024b; Wen et al., 2024; Xu et a [2024] Chen & Li
2024 Jeong al. 2024) attempted. Some of them (Wu et 20033 Wen et a 20224) perform
fine-graine RLHF using PPO, while our approach is based on the more efficient and cost-friendly
DPO. Most similar to ours, some approaches attempt mitigate hallucination using DPO, and their main difference lies in the different ways of constructing preference data. For example, Lin et al. (2024); Jeong et al. (2024) uses an external estimator or annotator to construct factual preference data, while Chen & Li (024)) Zhang et al. (2024b) rely on the model itself, and Tian et al. (2023) has both solutions. However, all of them only use the factuality of the entire response as the sparse reward signal, making factuality alignment less effective. In contrast, our approach implements fine-grained factuality alignment using sentence- reward signals.
Knowledge Editing. The primary cause of the hallucination is the fragility of parametric knowledge in LLMs (Wang et al.] 2024a). To keep knowledge truthfulness and reliability, many works attempt to edit it, including knowledge and representation editing. Knowledge editing (Zhang et al. 2024a]
Wang et al. 2024b] McAleese et al., (2024) focuses on selectively adjusting model parameters that store specific pieces of knowledge, while representation editing (Zou et al. 2023 Wu et al.) 2024) modifies how the model conceptualizes knowledge to update the information retained within LLMs.
Unlike them, our approach does not explicitly edit the knowledge-relatee weights or representations
but we hypothesize factuality alignment implicitly modifies the knowledge structure inside the model.
6 CONCLUSION
In this paper, we introduce a fine-grained factuality alignment method, Mask-DPO. Incorporating sentence-level factuality as reward signals, Mask-DPO accurately masks the encouraging signal of incorrect sentences in the preferred samples and the punishment signal from factual contents in the non-preferred samples, thus improving the effectiivneess of learning. Extensive experimental results demonstrate that Mask-DPO significantly increases the factuality of LLMs in both in-domain and out- of-domiin evaluation. The study on the generalization property of Mask-DPO using different training sample scaling strategies reveals that the diversity of topics is more important to the generalization than the number of questions, On the implication of this observation, we hypothesize that LLM learns a model-specific graph-like knowledge structure in training, which propagates the impact of factuality alignment based on the distance between topics, where nearby topics will also be implicitly aligned, even if they not seen during alignment. We conduct proof-of-concept experiments and show that factuality alignment indeed essentially aligns the internal knowledge of LLMs. We hope our study could pave the way for future research on scaling the factuality alignment of LLMSs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank the anonymous reviewers and area chair for their helpful comments. This project is supported by the Shanghai Artificial Intelligence Laboratory. The authors would like to thank Songyang Gao and Kuikun Liu for their valuable suggestions and comments.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302. 230.02330. 2023.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Zheng Cai. Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 2024.
Weixin Chen and Bo Li. Grath: Gradual self telthffuuti for large language models. arXiv preprint 1222: 2024.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183. 2024.
LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm.
https
//github. com InternIM/ Imdeploy 2023a.
XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1lm. https: http://gitubbb.. com/ InterntM/xtuner 2023b.
Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint arXiv:2303. 17574, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.21
2024.
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4453-4462, 2019.
Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah-v2: Scaling analytical hallucination annotation of large language models, arXiv preprint arXiv:2407.04000 04693, 2024.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0464 2024.
Minbyul Jeong. Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, and Jaewoo Kang. Olaph: Im- proving factuality in biomedical long-form question answering, arXiv preprint 11111 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2022.
11

---

Published as a conference paper at ICLR 2025
Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu, Bryan Wilie, Min Zeng, and Pascale Fung. RHO: Reducing hallucination in open-domain dialogues with knowledge grounding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 4504-4522, Toronto, Canada, July 2023. Association for Computational URL htt.tts:: / anthology........ org
Linguistics. doi: 10.18653/v1/2033 10.18663311122223ffdingg.. 2023. wwwdiignnnnnciiieeen 225
Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah: Analytical annotation of hallucinations in large language models. arXiv preprint MrXXiv2405.203/55 15, 2024.
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating
question answering in the era of large language models. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACI. 2023, Toronto, Canada, July 9- 14, 2023, pp- 2591-5606. Association for Computational Linguistics, 2023. doi: 00.18653//1/22223..2 307. URL https: pettps://doi.crgg /10 1...1.1..53 18e.186633//00 http://12222...
long. 307
Harrison Lee, Samrat Phatale, Hassan Mansoor. Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint wrXiv:2309.00263 2023.
Sheng-Chine Chich Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen-tau Yih, and Xilun Chen. Flame: Factuality alignment for large language models. arXiv preprint 2405:2240..15523 2024.
Nat McAleese, Rai Michael Pokorny Juan Felipe Ceron Uribe, Evgenia Nitishinskay,, Maja Trebacz, and Jan Leike. Llm crities help catch Ilm bugs. arXiv preprint arXiv:2407.00215,, 2024.
Tomas Mikolov. Efficient estimation of word representations vector space. arXiv preprint
arXiv: 3781, 2013.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zeitlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision long form text generation. In EMNLP, 2023. URLpttps::/ www.v.ow..............o....o /bs/
2305. 14251
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grad -ie-ggrieed hallucination detection and editing for language models. arXiv preprint ar 24110:2401.06855, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228. 2024.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledge- able are large language models (Ilm)? aka will 1mms replace knowledge graphs? arXiv preprint a00i 10168, 2023.
12

---

Published as a conference paper at ICLR 2025
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arX1n:2408.00118. 2024.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint wxxv:2311.0840/ 2023.
Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, et al. Knowledge mechanisms in large language models: A survey and perspective. arviv preprint arXiv:2407.15017 2024a.
Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Editing conceptual knowledge for large language models. arXiv preprint XXXv:2403.062550 2024b.
Jenry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang. Cosmo Du, et al. Long-form factuality in large language models. arXiv preprint arXiv:2403. 18802, 2024.
Xueru Wen, Xinyu Lu, Xinyan Guan, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. On-policy fine-grained knowledge feedback for hallucination mitigation. arXiv preprint 12Xiv:2406.12221,, 2024.
Rongxiang Weng, Heng Yu, Xiangpeng Wei, and Weihua Luo. Towards enhancing faithfulness for neural machine translation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2675-2684, 2020.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306. 01693, 2023.
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404.03592, 2024.
Hongshen Xu, Zichen Zhu, Da Ma, Situo Zhang, Shuai Fan, Lu Chen, and Kai Yu. Rejection improves reliability: Training Imss refuse unknown questions using from knowledge feedback. arXiv preprint 2rXiv:2403.183493 2024.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024a.
Kailai Yang. Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Se- lective preference optimization via token-level reward function estimation. arXiv preprint arXiv:2408.1 13518, 2024b.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 1:XX::2441..0022 2024.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404.1 1999, 2024.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang. Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.0m 2401::200000/866 01286, 2024a.
13

---

Published as a conference paper at ICLR 2025
Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation. arXiv preprint arXhv:2402.09267 2024b.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, al. Secrets of rlhf in large language models part Ppo. arXiv preprini arXiv:2307.04964 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach ai transparency. arXiv preprint arXiv:2310.01405, 2023.
A IMPLEMENTATION DETAILS
In our experimental framework, we adopt the 1-8-IstueB- (Dubey al. 2024 as the base model and the ANAH-v2 (Gu et al. 2024) as the fine-grained reward model to annotate factuality.
Preference data construction. We generate 32 candidate responses per question and selected the ones with the highest and lowest scores to form preference pairs after fine-grained factuality estimation. The decoding strategy involves the top-k (k = 40) sampling with a temperature of 0.8.
For each question, we sample multiple responses from the model. For response, we first split it into several sentences, and then annotate them sentence by sentence using the hallucination annotator. We choose ANAH-v2 (Gu et al., 2024) as the annotator, which provides four types of labels for each sentence, ie., 'No Facts', "No Hallucination'. 'Contradictory' and "Unverifiable.. Following the same setup in AN as hallucination. We take the proportion of sentences factuality score of that response. Based the factuality score, rank between multiple responses and select the highest-scoring and lowest-scoring responses to form preference pairs.
treat the first two types as hallucination-fie ffre and the last two
response that is hallucination-fres as the
Fine-grained factuality tuning. We train the base model with the following settings and hyper- parameters: the epoch is 3, the learning rate is 5e-6, the batch size is 64, and the AdamW optimizer is the cosine annealing learning rate scheduler.
Our model is trained on NVIDIA A100 GPUs. The training and inference frameworks are Xtuner (Contributors) 2023b) and LMDeploy (Contributors) 2023a) respectively.
Embedding, clustering, and choosing the far group and the near group. For each topic, we use the output of the last hidden layer of the model its embedding vector. The embedding vectors of the topics in the test set are treated as multiple cluster centroids. For each topic in the training set, we calculate the Euclidean distance from its embedding vector to each cluster centroid. Each training set topic is then assigned to the closest cluster centroid, forming a series of clusters. Within each cluster, the training set topics are further divided into two groups-fie far and nasrd on their distances to the cluster centroid. Topics closer to the centroid are grouped into the near group, while those farther away are grouped into the far group. Finally, the far groups from all clusters are merged into a single large far group, and similarly, the near groups are merged into single near group. Based on the group to which a topic belongs, preferred pairs are reorganized into two separate training datasets. These datasets are then trained separately.
We do not use any special prompts other than the hallucination annotations. Both in the construction of the preference data and in the evaluation process, directly use the question in the dataset as a prompt. For hallucination annotation in fine-grained preference data construction, we use the prompt from Gu et al. (2024), including factual existence judgment, reference information extraction, and hallucination-type judgment.
Prompt.
B ADDITIONAL EXPERIMENTS
In this section, provide additional experiments to further validate the effectiveness of Mask-DPP including comparisons with other methods ($ B.J., evaluations on standard benchmarks ($ B.2) and the impact of different inference hyper-parameter ($ B.3
14

---

Published as a conference paper at ICLR 2025
B.1 COMPARATIVE EXPERIMENTS
Comparision with SFT+PO. The default setting of Mask-DPO is to perform fine-grained preference learning on the instruct model. To provide greater transparency on the training procedure, we compare with performing SFT+PO on the base model. Specifically, we apply SFT the base model (not instruction-tunedl Llama3. 100ma..1-B.. 1-8B, using the chosen responses from the existing preference data. Subsequently, we perform Mask-DPO on the supervised model.
As shown in Table 7 executing SFT+Mask-DPO from the base model has similar results as executing Mask-DPO directly from the struet model, which shows that our fine-grained preference learning approach does not have any specific assumption reliance on the SFT process.
Table Comparision with SFT+PO.
# Correct
# Incorrect
% Score
Llama-3.1 SFT + Mask-DPO
631
192
76.67
Llama-3.1-8 1--ma-3.-8B-Insttreee 8B-Instruct Mask-DPP DPO
547.6
161.4
77.53
Comparision with other PO Strategies. We conduct comparative experiments TDPO (Zeng it al.
2024), DPOP (Pal et al. 2024). and SePO (Yang et al.) 2024b).
The specific configurations plementation, specifically the TDPO2 method with the alpha parameter set to 0.5. (2) DPOP: The lambda parameter is set to 50, consistent with the original paper. (3) SePO: We utilize the Oracle Model provided in their work to estimate the token-level reward function. For all experiments, we use 1--tmaa..--B--masee
follows: (1) TDPO:
We use the default settings from their im-
the base model and ANAH-v2 for evaluation.
As shown in Table g the performance of TDPO and DPOP is similar to DPO, while SePO performs significantly worse than DPO. This discrepancy may be due the Oracle Model used in SePO being trained on our tasks, which limits its ability to accurately estimate the token-level reward.
Table 8: Comparision with other PO Strategies.
# Correct
# Incorrect
% Score
DPO
446.4
206
68.
DPOP
593
291
67.08
TDPO
703.6
284
70.95
SePO
254
611.4
58.45
Mask-DPO
547.6
161.4
77.53
Comparision with factuuaiiy alignment methods. To help better position Mask-DPO in the context of the existing approach. except FactTune (Tian et al., 2023), we supplement the comparison with another state-of-the-art factuality alignment method, i.e., Flame (Lin et al.) 2024h. Specifically, we construct factuality preference data on our training set and perform SFT+DPO following the method described in Flame. We use Llama3. 1-8B-Innstrue as the base model and use ANAH-v2 for evaluation. Table 2 reports the comparison between Mask-DPO and the other two SOTA methods, where Mask-DPO gives better results.
Table Comparision with factuality alignment methods.
Correct
# Incorrect
% Score
FactTune
657
499
56.83
Flame
447
263
62.96
Mask-DPM
547.6
161.4
77.53
15

---

Published as a conference paper at ICLR 2025
B.2 EVALUATION ON STANDARD BENCHMARKS
To analyze the effect of factuality alignment over other LLM capabilities, we supplement the evaluation with relevant benchmarks for math and code. Specifically, we test the model on GSM8K, MATH, GPQA, Human Eval, and MBPP datasets using the OpenCompass evaluation framework. The base model Llama3.1188 --8B-Instt
Table 10 presents the results of the model these benchmarks before and after training. Mask-DPO shows a slight decrease in performance on one math benchmark (MATH) and two code benchmarks (Human Eval and MBPP), while demonstrating improvements two math benchmarks (GSM8K and GPQA). These results suggest that factuality alignment methods may have marginal effects on the other capabilities of the model.
Table 10: Evaluation on Standard Benchmarks.
GSM8K
MATH
GPQA
Human Eval
MBPP
Baseline
84.91
52.72
26.77
70.73 68.29
71.21
Mask-DPO
86.20
47.68
31.82
70.04
B.3 IMPACT OF DIFFERENT INFERENCE HYPPERPPARAMETE
To analyze the effect different inference ppprr-parameters i.e. temperature, we conduct series of experiments investigate the effect of this parameter performance. Specifically, we tested four temperature settings: 0.25, 0.5, 0.75, and 1.0. We exclude 0 because sampling at this value produced insufficiently diverse data for task, making it challenging to construct preference data. The other experimental conditions are consistent with those described in the paper. We use Llama3.1 1.8.1-188Innstte the base model and evaluate performance using ANAH-v2.
The comparison results are shown in the Table 11] Besides the default setting (0.8), the best performance is observed at temperature 0.75, while performance at temperature 0.25 is lower. Interestingly, this trend aligns with findings in the DPO paper (Rafailov et al., 2024), despite differences in task objectives. This suggests that the optimal sampling temperature for our task around 0.8.
Table 11: Evaluation on different inference setup.
Temperature
# Correct
Incorrect
% Score
0.25
485
259,2
65.30
0.5
539
244.8
67.96
0.75
733
219
76.02
1.0
732
240.6
74.24
0.8 (default)
547.6
161.4
77.53
C ADDITIONAL DISCUSSION
In this section, we provide additional discussions about the hypothesis ($ and the impact Mask-DPO on the quality of the response ($
C.1 we mentioned in 4.2
C.1 DISCUSSION ABOUT HYPOTHESIS
Affinity analysis. The idea-there is some structure in language modeling-has been explored in previous classic works (Mikolov et al. 2013 Mikolov 2013 Bengio et al., 2000), which argue that language models produce similar representations of words with high affinities. In other words, for an ideal language model, the relevance of the semantic space and the similarity of the semantic space represent spaces that consistent.
16

---

Published as a conference paper at ICLR 2025
Building on this understanding, we propose the internal knowledge structure hypothesis. As discussed in Section 4.2, we hypothesize that the model's blurred recognition of similar topics leads to the mixed use of their respective subordinate content, resulting in phenomena akin to hallucination (cf. Eq 8). We conceptualize factual alignment as a process of adjusting the relationships between topics. During the alignment of given topic, the affinity between the topic and its subordinate content increases, while the affinity with the subordinate content of similar topics decreases, This ultimately manifests as reduction in hallucinated content.
We randomly selected 200 data from the preference dataset used for training to perform a quantitative analysis of affinities. Specifically, for each preference data point, we extracted the entities from both the chosen response and the rejected response, embedding these entities using word embeddings. We then calculated the average distance between the word vectors these entities and the vector of the topic to which the preference data belongs. This evaluation was performed on the model separately before and after training. The results indicate that, after training. the average distance between the topic vector and the entities in the chosen response decreased by 1.97, while the average distance to the entities in the rejected response increased by 4.09. These findings support the reliability of our fffinity-basedd explanation of the internal knowledge structure.
Discussion about the question-based knowledge structure. For the hypothesis about the internal knowledge structure, while Corollary in Section 4.3 suggests that the structure may be topic- based, does not rule out the possibility of a question-basee structure. To address this, we designed an additional experiment inspired by Corollary I's setup, focusing clustering based questions. as the base and embedding model for evaluation, we analyzed the results.
Using Llama3. 1-......-8--Iteiiee
As shown in Table 12, the clustering outcomes based questions and topies different, indicating that the knowledge structure is not question-based.
significantly
Table 12: Evaluation question- based clustering strategies for preference data construction.
Distance # Correct
# Incorrect
% Score
Diff
far near
541.4 518
214 195
71.69 71.92
0.23
C.2 DISCUSSION ABOUT THE QUALITY OF THE RESPONSE
In order analyze whether Mask-DPO affects the quality of the model-generated responses, e.g. overall consistency and completeness, we add text quality evaluation. We use the GPT-2 to compute the average perplexity of text generated by the base model (Llama3.1-8B --stttrructt Instruct) and the model after Mask-DPO training. Evaluated on the ANAH test set, the average PPL of the base model is 44.20 while after Mask-DPO training the average PPL. is 34.91, suggesting that the training does not affect the quality of the generated text, and even makes it better.
D CASE STUDY
To analyze the effect of our method, we provide a case study about the generated response before and after Mask-DPO. As shown in Figure 3 there is a significant decrease in the number of phantom parts in the responses after Mask-DPO training, illustrating the effectiveness of our method.
E LIMITATION
Although this study presents a novel and effective fine-grained factuality alignment framework, there are some limitations. Despite Mask-DPO's performance in fine hallucination relief is remarkable, remains somewhat exaggerated. Because the model we used to construct the preference data is the same as the one used for the review, this could pose a potential risk reward hacking. Although introduced additional evaluation models to show that this hacking phenomenon is not significant, still cannot be completely avoided. Furthermore, we only perform evaluation on ANAH and Biography. However, these datasets might not encompass the full spectrum of real-world scenarios
17

---

Published as a conference paper at ICLR 2025
where hallucinations pose problem. And this work primarily uses Llama3. -----tstttut and as the backbone of the base model. Other different underlying models and different numbers of parameters not explored. Lastly, sentence-level hallucination annotations not fine-grained enough. Notely, the form of Mask-DPO itself is general and does not assume that only sentence-level annotations be used. The results in the paper show that using finer-grained annotations, such as sentence-level annotations, can lead to better results than using response-level annotations. If external hallucination annotator can provide more detailed annotations, such as the level of clauses, noun phrases, verb phrases, etc., we believe the results will be better. However, there is currently lack of hallucination annotators that can provide more fin-graine information.
18

---

Published as a conference paper at ICLR 2025
Question:
What is hei-tiki?
Reference Document:
Hei-tiki, small neck pendant in the form of human fetus, used by the Mäori New Zealand a fertility symbol. Usually carved of green nephrite or a jadelike stone called pounamu that is found along the western coast of the South Island, hei-tikis normally are worn only by women. The object is believed possess magical powers that increase as is passed on from generation to generation. According to one idea, the hei-tiki protects its wearer against the vengeful spirits of stillborn infants, who have been deprived of the chance to live. Another theory holds that the figure represents the Polynesian god Tiki, the creator of life. Hei-tikis have been prized by European and American collectors for their beauty and elegance, but to the Mãori the greatest value of these pendants lies their possession of magical powers and in the prestige acquired from previous owners.
Baseline:
A hei-tiki is traditional Mãori pendant that originated in New Zealand. is significant cultural artifact that holds spiritual and historical importance for the indigenous Mãori people.
A hei-tiki is carved greenstone (pounamu) pendant in the shape of a human figure, typically woman
child. The pendant is usually made from single piece of greenstone, which is a type of jade that is highly valued in Mãori culture. The carving often intricate and features a range of details, including facial features, hair, and clothing. Hei-tiki pendants were traditionally worn by Mãori women. They were often passed down through generations as family heirlooms and were believed possess spiritual energy and protection. The pendant was thought to bring good fortune, health, and prosperity to the wearer. In Mãori culture, hei-tiki pendants were also associated with the concept of Whakapapa,ór genealogy. The pendant was seen connection to one's 's ancestors and the spiritual realm.
Today, hei-tiki pendants are highly valued for their cultural significance and are often worn as a symbol of Mãori identity and pride. They are also highly sought after by collectors and are considered significant part of New Zealand's cultural heritage.
Mask-DPO:
A hei-tiki is traditional Mãori pendant. It is a distinctive and iconic piece of Mãori jewelry that often associated with the Mãori people's rich history and cultural heritage. Here are some key facts about hei-tiki: wh is hei-tiki?** A hei-tiki a carved pendant that typically depicts child figure.
**Significance:** ** Hei-tiki considered a treasured and sacred object in Mãori culture, often passed down through generations family heirloom. 3. **Purpose:** The exact purpose of hei-tiki is not well-documented, but is believed have been 4. **Cultural significance: Hei-tiki is an important part of Mãori cultural heritage and is often seen as a representation of the Mãori people's 's rich history and traditions, In summary, hei-tiki is a significant and sacred object in Mãori culture, often associated with the people's history, traditions, and cultural heritage. I hope this helps you understand what a hei-tiki is!
symbol of status, wealth.
Figure 3: The case study about the generated responses before and after Mask-DPO. Here, "Baseline" denotes the response from the model before Mask-DPO, i.e. Llama3.1-8B-Instruus "Mask- DPO" denotes the response from the model after Mask-DPO. In generations, we use blue color to present the hallucination content.
19


Reviews:
[{"review_id": "uf0GGFpDnJ", "reviewer": "Reviewer_bLBp", "summary": "In this paper, the authors introduce an alignment method to improve factuality in LM responses.\nThe method, based on DPO, leverages factuality annotations at the sentence level to avoid optimizing non-factual sentences in chosen responses and to avoid penalizing factually correct sentences in rejected responses during preference optimization.\nExperiments in and out of domain demonstrate the effectiveness of the strategy on improving response factuality.\nMost intriguing, the authors provide a hypothesis, accompanied with preliminary experiments, about how factuality alignment influence topic representation and the generalization capabilities of an LM.", "questions": "## Appendix A\n-\tRegarding the inference setup, temperature=0.8, was hyper-parameter tuning performed? Ideally, it would be helpful to show how performance varies across temperature values (e.g. 0, 0.25, 0.5, 0.75, 1.0), as it is done in PO literature.\n\n## Granularity of Annotation\nIn the paper, annotation is gathered at the sentence level. Would it be possible to apply this same annotation procedure to more fine-grained syntactic units? For instance, clauses, noun phrases, verb phrases. Do you have an intuition as to how impactful such a setup would be?\n\n## Reporting results\n-\tWas a statistical significance test performed on the reported metric results?\n-\tI am not familiar with ANAH and FactScore metrics, would it be safe to assume they follow a gaussian distribution? If not, it is important to consider that, in cases where the metric score distribution cannot be considered gaussian, it is better to report results over bootstrap resampling of the metric scores. Please refer to EleutherAI’s Evaluation Harness (lm-eval), a popular evaluation library for LLMs, for details.\n\n[4] https://github.com/EleutherAI/lm-evaluation-harness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The proposed method is well presented and supported mathematically.", "weaknesses": "W1. Even though the experimental setup includes a wide range of model families, it does not include transparent SFT+PO (instruction tuning + preference optimization) training setups, training PO directly over instruct models. Previous work in PO encourages starting from a base model (not instruction-tuned), perform SFT over chosen responses, and then perform PO. Such setup provides high transparency on the training procedure of a final chat / PO model. For reference, please refer to the DPO paper.\n\nW2. There is a lack of comparison against relevant, recent PO strategies. Similar work that implement some form of masking or granularity below response are: TDPO, DPOP, SePO, to name a few.\n\nW3. Whilst the paper presents extensive results on factuality metrics, a more comprehensive evaluation on standard benchmarks is needed. In this way, we could appreciate the effect of factuality alignment over other LLM capabilities, e.g. mathematical reasoning or harmful generation.\n\n[1] https://arxiv.org/abs/2404.11999\n[2] https://arxiv.org/pdf/2402.13228\n[3] https://arxiv.org/pdf/2408.13518", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m9BkiYqAzW", "reviewer": "Reviewer_VhTU", "summary": "This paper proposes a novel fine-grained factuality alignment method called Mask-DPO to mitigate hallucinations in large language models (LLMs). Mask-DPO incorporates sentence-level factuality information as a mask signal to guide the preference learning, avoiding the reward for incorrect sentences in preferred samples and the penalty for factual sentences in non-preferred samples. Experiments demonstrate that Mask-DPO significantly improves the factuality of LLM responses on both in-domain and out-of-domain datasets, even when the test questions and topics are unseen during training. The authors also study the generalization property of Mask-DPO and find that scaling the number of topics in the training data is more effective than scaling the number of questions. They hypothesize an internal knowledge graph learned by LLMs and provide proof-of-concept experiments to verify it.", "questions": "Section 3.1: It would be helpful to provide more details on the specific subset of the Biography dataset used for out-of-domain evaluation, such as the criteria for selecting the questions and the diversity of the topics covered.\n\nSection 3.2: The authors could consider adding a case study or qualitative analysis of the generated responses before and after applying Mask-DPO. This would give readers a more intuitive understanding of how Mask-DPO improves factuality at the sentence level.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Mask-DPO addresses a critical issue of hallucination in LLMs, which hinders their practical applications in various domains. The proposed method provides a more effective solution compared to existing response-level preference learning approaches by leveraging fine-grained sentence-level factuality signals.\n\n2. The paper provides valuable insights into the generalization property of Mask-DPO by studying different training data scaling strategies. The finding that scaling the number of topics is more effective than scaling the number of questions sheds light on how to efficiently construct training data for factuality alignment.\n\n3. The hypothesis of an internal knowledge graph learned by LLMs during pre-training is intriguing and opens up new research directions. The authors provide proof-of-concept experiments to support this hypothesis, such as ablating different topic sampling strategies and comparing the factuality of best-of-N responses before and after alignment.", "weaknesses": "1. While the paper compares Mask-DPO with DPO, it would be beneficial to include a more comprehensive comparison with other state-of-the-art factuality alignment methods. This would help to better position Mask-DPO in the context of existing approaches and highlight its unique advantages.\n\n2. The hypothesis of the internal knowledge graph learned by LLMs is intriguing but lacks sufficient theoretical justification and empirical evidence. The authors could provide a more rigorous theoretical analysis of how the knowledge graph is formed during pre-training and how it relates to the factuality alignment process. Additionally, visualizing the graph structure or quantitatively measuring the affinity between nodes would further support the hypothesis.\n\n3. The paper would benefit from a more thorough proofreading to fix minor typos and grammatical errors (e.g., \"anachronistic sentences\" in Figure 2 caption, \"preferece\" in line 169 of Section 2.1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WT6D8BAVbK", "reviewer": "Reviewer_upqN", "summary": "It proposes Mask-DPO, a fine-grained factuality alignment method, which performs a sentence-level masking to denoise inaccurate training signals inherent from noisy sentences by utilizing external factuality evaluator. Conducting experiments with various base models, Mask-DPO outperforms vanilla-DPO and a existing factuality alignment method (FactTune) in both in-domain and out-of-domain datasets. Furthermore, it explores the generalization property of the method, which hypothesizes two corollaries to prove the characteristics of factuality alignment achieved by MASK-DPO.", "questions": "[missing references] Long-form factuality generation in related work\n\n1) Long-form Factuality in Large Language Models (Wei et al., 2024)\n2) OLAPH: Improving Factuality in Biomedical Long-form Question Answering (Jeong et al., 2024)\n\n[Sampling strategy] In line 344, it is quite hard to understand the first sentence that explains about the non-policy model effectiveness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Reasonable Analysis of Factuality Alignment Through Data Scaling**\n\nThe study provides a well-founded hypothesis on the effects of factuality alignment by comparing outcomes based on the number of topics and questions. The subsequent proof-of-concept experiments offer a valid approach to support this hypothesis.\n\n**Novelty of provided Methods\"\n\nThe authors provide a novel mask-DPO method which hinders to learn unfactual text fragments lied in each sentence.", "weaknesses": "**Potential for Errors Due to Sentence-Level Annotation**\n\nThe paper ranks responses through sentence-level evaluation, but this method may not always accurately reflect factuality. Depending on the type of sentence or factual argument, sentence-level evaluation may misalign with actual factual accuracy. For instance, consider two responses conveying the same information:\n        - Response 1: Consists of 10 short sentences.\n        - Response 2: Consists of 3 long sentences.\nIf hallucination occurs in only one part of each response, the annotation scores could vary significantly (e.g., 9/10 vs. 1/3). Factors such as sentence segmentation and fact placement can impact response ratings, introducing potential errors in sentence-level evaluation.\n\n**concerns about overall quality of the responses**\n\nWhen it comes to response coherence, it raises concerns that the quality of the response may be compromised when certain sentences are masked. While masking can improve factual accuracy by filtering out incorrect signals from sampled responses, it risks disrupting the overall coherence and completeness of the response, as preference learning is based on responses where specific sentences have been removed partway through. Therefore, it may potentially leads to unexpected or lower-quality outputs even if they are factually correct. I understand that the primary focus of the paper is on the factuality of model responses, yet factuality itself aims to enhance overall response quality in LLMs.  Given this, I wonder if there is any evaluation or consideration of how the sentence removal approach in Mask-DPO's preference data construction might negatively impact response quality.\n\n**insufficient details**\n\n1) Ranking and Pairwise selection process\n2) Prompts\n3) Section 4.3: Proof of concept\n    - embedding details\n    - clustering topic vectors\n    - choosing the far group and the near group", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1sZ0DmKkOd", "reviewer": "Reviewer_GYLY", "summary": "This work presents a method for training models via preference optimization in order to produce more factually correct responses and as a result mitigate halluciantions. This is achieved by (1) creating preference data where chosen responses are more factually correct than dispreferred ones and (2) by masking parts of each response if they contain inacccurate information. \nTo realise the above, the authors use an external sentence-level factual evaluator to measure the correctness of each sentence inside each response and consider the ratio of correct sentences as the preference score. Then, a binary mask is employed into the DPO loss thus increasing the probability of correct facts and decreasing the probability of incorrect facts.\n\nResults on in- and out-of-domain datasets suggest that the resulting models show significantly better performance across two different evaluators.\nIn addition, the authors hypothesize that such an alignment impacts the internal knowledge of LLMs, which they showcase via clustering responses on specific topics and best-of-N responses. Topics close to each other have better factuality.", "questions": "Questions:\n- Line 262: You mention that the models are post-trained versions, what do you mean by that? That the models are instruction tuned?\n- I am kinda missing the point of the ablation in Table 3. Do you aim to show that the quality of responses depends on the underlying model used to generate them? In this setting, the data are generated by model X but LLaMA is trained with Mask-DPO? I think it is already proved that the model needs to be in-distribution for DPO to work (original DPO paper) - which explains why llama3 works best in this case too?\n\nComments:\n- Line 219: inaccuracies facts -> incaccurate facts\n- Line 283: un-preferred -> dispreferred (in other places as well)\n- Line 323: rewarding hackers -> reward hacking\n- Table 4: what does the +X refer to? which is the baseline score?\n- I feel the name \"Mask-DPO\" is quite misleading as it is very generic and it is not directly associated with factuality.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a simple and straighforward method to improve models to produce more factually correct responses\n- Results on the evaluated benchmarks indicate significant improvements\n- Proof of concept experiments are designed to explain the performance of the models and the effect that factual alignment has in their internals", "weaknesses": "- One could argue that even in a single sentence, there can be factually incorrect information. For example, if the sentence is long enough, different clauses can contain conflicting information. How would you be able to mitigate this without completely ignoring the entire sentence?\n- The method looks general enough, however it is targeted on factuality alone. Do you think if you replaced factual checking with cohesiveness / fluency / some other metric that is easy to obtain, models could also improve towards that direction too?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LIbrh3lRtJ", "reviewer": "Reviewer_AbTR", "summary": "The paper presents a fine-grained factuality tuning method, referred to as Mask-DPO, designed to incorporate sentence-level factuality. Extensive experiments demonstrate the effectiveness on both in-domain and out-of-domain datasets. It also provides an analysis from two dimensions: the number of different topics and the diversity of the questions.", "questions": "Please see weakness.\n\nAdditionally, in Table 5, it’s intriguing that using embeddings from other models seems to make the differences less distinct. Why alternative embeddings are used instead of relying solely on those from the policy model itself?\n\nI would like to increase my score if my concerns are addressed properly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The approach of filtering out false sentences within preference pairs is particularly intriguing, as it helps refine the dataset by ensuring that only factual information is prioritized.\n\n2.The paper establishes well-founded hypotheses regarding the functionality of the factual alignment process and conducts experiments to validate these assumptions.", "weaknesses": "1.Mask-DPO operates at the sentence-level granularity, employing a sentence-level hallucination annotator to mask undesired sentences within training preference pairs. However, it's worth noting that many responses mix true and false information in a single sentence. I'm curious how Mask-DPO handles these cases since pulling apart accurate details from inaccuracies can be tricky. Clarifying this might help us see how well the model deals with more complex responses.\n\n2.The experiments seem focused on long-form generation, which raises the question of how the model would perform on shorter, QA-style tasks.\n\n3.The proof of Corollary 2 in Section 4.3 is less convincing. How did you arrive at this knowledge structure corollary based on the hypothesis you mentioned earlier? Specifically, if the model-specific graph were focused on questions rather than topics, would we still arrive at the same conclusions? Please clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)