LongPO: Long Context Self-Evolution of Large Language Models through
  Short-to-Long Preference Optimization
Authors: name: Guanzheng Chen, name: Xin Li, name: Michael Qizhe Shieh, name: Lidong Bing

Large Language Models (LLMs) have demonstrated remarkable capabilities
through pretraining and alignment. However, superior short-context LLMs may
underperform in long-context scenarios due to insufficient long-context
alignment. This alignment process remains challenging due to the impracticality
of human annotation for extended contexts and the difficulty in balancing
short- and long-context performance. To address these challenges, we introduce
LongPO, that enables short-context LLMs to self-evolve to excel on long-context
tasks by internally transferring short-context capabilities. LongPO harnesses
LLMs to learn from self-generated short-to-long preference data, comprising
paired responses generated for identical instructions with long-context inputs
and their compressed short-context counterparts, respectively. This preference
reveals capabilities and potentials of LLMs cultivated during short-context
alignment that may be diminished in under-aligned long-context scenarios.
Additionally, LongPO incorporates a short-to-long KL constraint to mitigate
short-context performance decline during long-context alignment. When applied
to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully
retains short-context performance and largely outperforms naive SFT and DPO in
both long- and short-context tasks. Specifically, \ourMethod-trained models can
achieve results on long-context benchmarks comparable to, or even surpassing,
those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context
annotation and larger parameter scales.