Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
Authors: name: Yuzhe Gu, name: Wenwei Zhang, name: Chengqi Lyu, name: Dahua Lin, name: Kai Chen

Large language models (LLMs) exhibit hallucinations (i.e., unfaithful or
nonsensical information) when serving as AI assistants in various domains.
Since hallucinations always come with truthful content in the LLM responses,
previous factuality alignment methods that conduct response-level preference
learning inevitably introduced noises during training. Therefore, this paper
proposes a fine-grained factuality alignment method based on Direct Preference
Optimization (DPO), called Mask-DPO. Incorporating sentence-level factuality as
mask signals, Mask-DPO only learns from factually correct sentences in the
preferred samples and prevents the penalty on factual contents in the not
preferred samples, which resolves the ambiguity in the preference learning.
Extensive experimental results demonstrate that Mask-DPO can significantly
improve the factuality of LLMs responses to questions from both in-domain and
out-of-domain datasets, although these questions and their corresponding topics
are unseen during training. Only trained on the ANAH train set, the score of
Llama3.1-8B-Instruct on the ANAH test set is improved from 49.19% to 77.53%,
even surpassing the score of Llama3.1-70B-Instruct (53.44%), while its
FactScore on the out-of-domain Biography dataset is also improved from 30.29%
to 39.39%. We further study the generalization property of Mask-DPO using
different training sample scaling strategies and find that scaling the number
of topics in the dataset is more effective than the number of questions. We
provide a hypothesis of what factual alignment is doing with LLMs, on the
implication of this phenomenon, and conduct proof-of-concept experiments to
verify it. We hope the method and the findings pave the way for future research
on scaling factuality alignment.