LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
  Critique-augmented Stepwise Information
Authors: name: Bowen Ping, name: Jiali Zeng, name: Fandong Meng, name: Shuo Wang, name: Jie Zhou, name: Shanghang Zhang

Long-form generation is crucial for academic writing papers and repo-level
code generation. Despite this, current models, including GPT-4o, still exhibit
unsatisfactory performance. Existing methods that utilize preference learning
with outcome supervision often fail to provide detailed feedback for extended
contexts. This shortcoming can lead to content that does not fully satisfy
query requirements, resulting in issues like length deviations, and diminished
quality. In this paper, we propose enhancing long-form generation by
incorporating process supervision. We employ Monte Carlo Tree Search to gather
stepwise preference pairs, utilizing a global memory pool to maintain
consistency. To address the issue of suboptimal candidate selection, we
integrate external critiques to refine and improve the quality of the
preference pairs. Finally, we apply step-level DPO using the collected stepwise
preference pairs. Experimental results show that our method improves length and
quality on long-form generation benchmarks, with almost lossless performance on
general benchmarks across various model backbones.