Learning to Reason for Factuality
Authors: name: Xilun Chen, name: Ilia Kulikov, name: Vincent-Pierre Berges, name: Barlas Oğuz, name: Rulin Shao, name: Gargi Ghosh, name: Jason Weston, name: Wen-tau Yih

Reasoning Large Language Models (R-LLMs) have significantly advanced complex
reasoning tasks but often struggle with factuality, generating substantially
more hallucinations than their non-reasoning counterparts on long-form
factuality benchmarks. However, extending online Reinforcement Learning (RL), a
key component in recent R-LLM advancements, to the long-form factuality setting
poses several unique challenges due to the lack of reliable verification
methods. Previous work has utilized automatic factuality evaluation frameworks
such as FActScore to curate preference data in the offline RL setting, yet we
find that directly leveraging such methods as the reward in online RL leads to
reward hacking in multiple ways, such as producing less detailed or relevant
responses. We propose a novel reward function that simultaneously considers the
factual precision, response detail level, and answer relevance, and applies
online RL to learn high quality factual reasoning. Evaluated on six long-form
factuality benchmarks, our factual reasoning model achieves an average
reduction of 23.1 percentage points in hallucination rate, a 23% increase in
answer detail level, and no degradation in the overall response helpfulness.