TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
Authors: name: Zhepei Wei; name: Xiao Yang; name: Kai Sun; name: Jiaqi Wang; name: Rulin Shao; name: Sean Chen; name: Mohammad Kachuee; name: Teja Gollapudi; name: Tony Liao; name: Nicolas Scheffer; name: Rakesh Wanga; name: Anuj Kumar; name: Yu Meng; name: Wen-tau Yih; name: Xin Luna Dong
Published: 2025-09-30 04:25:17

Abstract: While large language models (LLMs) have demonstrated strong performance on
factoid question answering, they are still prone to hallucination and
untruthful responses, particularly when tasks demand information outside their
parametric knowledge. Indeed, truthfulness requires more than accuracy --
models must also recognize uncertainty and abstain when unsure to avoid
hallucinations. This presents a fundamental challenge for existing methods:
approaches that optimize for accuracy often amplify hallucinations, while those
that encourage abstention can become overly conservative, sacrificing correct
answers. Both extremes ultimately compromise truthfulness. In this work, we
present TruthRL, a general reinforcement learning (RL) framework that directly
optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using
GRPO with a simple yet effective ternary reward that distinguishes correct
answers, hallucinations, and abstentions. It incentivizes models to reduce
hallucinations not only by providing correct responses, but also by enabling
abstention when uncertain, thereby improving truthfulness. Extensive
experiments across four knowledge-intensive benchmarks show that, compared to
vanilla RL, TruthRL significantly reduces hallucinations by 28.9% and improves
truthfulness by 21.1%, with consistent gains across various backbone models
(e.g., Qwen, Llama) under both retrieval and non-retrieval setups. In-depth
ablation study demonstrates that vanilla accuracy-driven methods, such as
supervised fine-tuning or RL with a binary reward, struggle to balance factual
correctness and uncertainty. In contrast, our proposed truthfulness-driven
TruthRL achieves strong performance in both accuracy and truthfulness,
underscoring the importance of learning objective design for developing
truthful LLMs.