LongReward: Improving Long-context Large Language Models with AI
  Feedback
Authors: name: Jiajie Zhang; name: Zhongni Hou; name: Xin Lv; name: Shulin Cao; name: Zhenyu Hou; name: Yilin Niu; name: Lei Hou; name: Yuxiao Dong; name: Ling Feng; name: Juanzi Li
Published: 2024-10-28 17:50:42

Abstract: Though significant advancements have been achieved in developing long-context
large language models (LLMs), the compromised quality of LLM-synthesized data
for supervised fine-tuning (SFT) often affects the long-context performance of
SFT models and leads to inherent limitations. In principle, reinforcement
learning (RL) with appropriate reward signals can further enhance models'
capacities. However, how to obtain reliable rewards in long-context scenarios
remains unexplored. To this end, we propose LongReward, a novel method that
utilizes an off-the-shelf LLM to provide rewards for long-context model
responses from four human-valued dimensions: helpfulness, logicality,
faithfulness, and completeness, each with a carefully designed assessment
pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
effectively improve long-context SFT models. Our experiments indicate that
LongReward not only significantly improves models' long-context performance but
also enhances their ability to follow short instructions. We also find that
long-context DPO with LongReward and conventional short-context DPO can be used
together without hurting either one's performance.