Teaching Large Language Models to Maintain Contextual Faithfulness via
  Synthetic Tasks and Reinforcement Learning
Authors: name: Shuzheng Si, name: Haozhe Zhao, name: Cheng Gao, name: Yuzhuo Bai, name: Zhitong Wang, name: Bofei Gao, name: Kangyang Luo, name: Wenhao Li, name: Yufei Huang, name: Gang Chen, name: Fanchao Qi, name: Minjia Zhang, name: Baobao Chang, name: Maosong Sun

Teaching large language models (LLMs) to be faithful in the provided context
is crucial for building reliable information-seeking systems. Therefore, we
propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
both short-form and long-form generation tasks without human annotations.
Specifically, we first synthesize short-form question-answering (QA) data with
four diverse tasks to construct high-quality and easily verifiable training
data without human annotation. Also, we propose Dual-GRPO, a rule-based
reinforcement learning method that includes three tailored rule-based rewards
derived from synthesized short-form QA data, while simultaneously optimizing
both short-form and long-form response generation. Notably, Dual-GRPO
eliminates the need to manually label preference data to train reward models
and avoids over-optimizing short-form generation when relying only on the
synthesized short-form QA data. Experimental results show that CANOE greatly
improves the faithfulness of LLMs across 11 different downstream tasks, even
outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.