Teaching Large Language Models to Maintain Contextual Faithfulness via
  Synthetic Tasks and Reinforcement Learning
Authors: name: Shuzheng Si; name: Haozhe Zhao; name: Cheng Gao; name: Yuzhuo Bai; name: Zhitong Wang; name: Bofei Gao; name: Kangyang Luo; name: Wenhao Li; name: Yufei Huang; name: Gang Chen; name: Fanchao Qi; name: Minjia Zhang; name: Baobao Chang; name: Maosong Sun
Published: 2025-05-22 10:10:07

Abstract: Teaching large language models (LLMs) to be faithful in the provided context
is crucial for building reliable information-seeking systems. Therefore, we
propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
both short-form and long-form generation tasks without human annotations.
Specifically, we first synthesize short-form question-answering (QA) data with
four diverse tasks to construct high-quality and easily verifiable training
data without human annotation. Also, we propose Dual-GRPO, a rule-based
reinforcement learning method that includes three tailored rule-based rewards
derived from synthesized short-form QA data, while simultaneously optimizing
both short-form and long-form response generation. Notably, Dual-GRPO
eliminates the need to manually label preference data to train reward models
and avoids over-optimizing short-form generation when relying only on the
synthesized short-form QA data. Experimental results show that CANOE greatly
improves the faithfulness of LLMs across 11 different downstream tasks, even
outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.