LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
Authors: Guanzheng Chen
Xin Li
Michael Shieh
Lidong Bing
Venue: ICLR 2025 Poster (2025)
Keywords: Long Context LLMs and Alignment
Large Language Models
Preference Optimization
Self-Evolution of LLMs
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A new long-context alignment method that enables LLMs self-evolve to excel on long-context tasks without annotated data and short-performance drops.

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities through pretraining and alignment. However, superior short-context LLMs may underperform in long-context scenarios due to insufficient long-context alignment. This alignment process remains challenging due to the impracticality of human annotation for extended contexts and the difficulty in balancing short- and long-context performance. To address these challenges, we introduce LongPO, that enables short-context LLMs to self-evolve to excel on long-context tasks by internally transferring short-context capabilities. LongPO harnesses LLMs to learn from self-generated short-to-long preference data, comprising paired responses generated for identical instructions with long-context inputs and their compressed short-context counterparts, respectively. This preference reveals capabilities and potentials of LLMs cultivated during short-context alignment that may be diminished in under-aligned long-context scenarios. Additionally, LongPO incorporates a short-to-long KL constraint to mitigate short-context performance decline during long-context alignment. When applied to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully retains short-context performance and largely outperforms naive SFT and DPO in both long- and short-context tasks. Specifically, LongPO-trained models can achieve results on long-context benchmarks comparable to, or even surpassing, those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context annotation and larger parameter scales. Our code is available at https://github.com/DAMO-NLP-SG/LongPO.

Published as a conference paper at ICLR 2025
LONGPO: LONG CONTEXT SELF-EVOLUTION OF LARGE LANGUAGE MODELS THROUGH SHORT-TO- LONG PREFERENCE OPTIMIZATION
Guanzheng Chen 1,2,3
Xin Li
Michael Qizhe Shieh
Lidong Bing"
National University of Singapore 2DAMO Academy, Alibaba Group
Hupan Lab, 310023, Hangzhou, China
Shanda AI Research Institute
ge chen@u. nus edu, xinting. 1etalxxlabbbba -nnc com michae www.hhshhhhhhpppee nus nus sg, lidong
com
ABSTRACT
Large Language Models (LLMs) have demonstrated remarkable capabilities through pretraining and alignment. However, superior short-context LLMs may underperform in long-context scenarios due to insufficient long-context alignment. This alignment process remains challenging due to the impracticality of human annotation for extended contexts and the difficulty in balancing short- and long- context performance. To address these challenges, we introduce LongPO, that enables short-context L.LMs self-evolve excel long-context tasks by in- ternally transferring short-context capabilities. LongPO harnesses L.MSs to learn from self-generated short-to-long preference data, comprising paired responses generated for identical instructions with long-context inputs and their compressed short-context counterparts, respectively. This preference reveals capabilities and potentials of LLMs cultivated during short-context alignment that may be dimin- ished in under-aligned long-context scenarios. Additionally, LongPO incorporates a short-to-lone KI. constraint to mitigate short-context performance decline during long-context alignment. When applied to Di--rall7B--Istuuct--- from 128K 512K context lengths, LongPO fully retains short-contex, performance and largely outperforms naive SFT and DPO both long- and short-context tasks. Specifi- cally, LongPO-trained models achieve results long-context benchmarks comparable to, or even surpassing, those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context annotation and larger parameter scales, Our code is available at hitts:://githne com/DANC AAMO-NNP-- SG/ SG/LengPO)
INTRODUCTION
Recent advancements in Large Language Models (LLMs) have revealed remarkable capabilities through extensive pretraining and subsequent alignment with human intentions. The alignment pro- cess, including methods such as Supervised Fine-Tuning (SFT) (Wei et 2022). Direct Preference Optimization (DPO) (Rafailov et al. (023). and Reinforcement Learning from Human Feedback (RLHF) (Christiano et al.) 2017 Ouyang et al.) 2022 Stiennon et al. 2020), has effectively un- leashed the potential of LLMs acquired during pretraining to achieve desired behaviors.
Although off-the-shelf alignment methods have made significant strides in short-context settings, their application long-context situations remains challenging (Bai et al. 2024). First, the scarcity of high-quality, long-context annotated data poses significant hurdle. Human annotation becomes impractical and less-reliable context length increases (Dubey et al. 2024), while synthetic data generation using advanced LLMs lacks scalability and remains resource-intedd ply concatenating existing short-context datasets has been shown yield unsatisfactory long-context performance (Liu et al.) 2024b). Second, long-context alignment methods grapple with the balance
Moreover, sim-
'This work was done during the internship of Guanzheng Chen at Alibaba DAMO Academy. Corresponning Author.

---

Published as a conference paper at ICLR 2025
Long-Conted Feaformance
Pottral7B-Lonn.P-1123
LLaMAS---BB-Instree--222
35
CL----9--Cha--M
GPT-4-128K
30
GL-M-4-9-Chht-122
+25.45
25
Misstral-BB--nsttrutt-22222
Short-Context Performance
Figure 1: The comparison of long-context (ooBench) and short-context (MMLU) performance among GPT-4-128K and smaller LLMs.
between preserving short-context proficiency and cultivating long-context capabilities (Liu et al, 2024b). For instance, the LLaMA-3.1 series incorporate merely 0.1% long-context data with over 99% short-context data during alignment to maintain the short-context performance (Liu et al.)
2024b). This limited exposure to natural long-context data may result in insufficient alignment, potentially blocking the intrinsic long-context capabilities in LLMs.
The challenges of long-context alignment suggest that the full potential of LLMs may remain tapped for long-context tasks. As illustrated in Figure I, even superior models such as GPT-4, which excel in short-context tasks, unexpectedly underperform in long-context scenarios. Interest- ingly, despite the much stronger short-context capabilities, GPT-4 still inferior to LLaMA3. 1-8B on long-context tasks. This disparity underscores the need for more effective long-context alignment methods to fully unleash the intrinsic power of LLMs across variable context lengths.
In this work, posit that the capabilities, deeply ingrained during short-context pretraining and alignment, can be effectively transferred to longer contexts without external guidance. To this end, we introduce Short-to-Long Preference Optimization (LongPO). steer long-context alignment by injecting internal short-context preferences into long-context scenarios. Specifically, we propose construct the preference data pairs by prompting the short-context LLM (e.g.. Mistral-Instrun instruction over a long document and, (2) short
with two inputs: (1) long input comprising input with the identical instruction over the relevant shortened chunk within the same document. We then designate the responses short and long inputs as chosen and rejected responses, respec- tively. The short-to-long preference, i.e., the discrepancies between each paired response, reveal the capabilities and potentials cultivated during short-context alignment that may be diminished in mnder-aligned long-contex scenarios. In order to bring forward the established capabilities, LongPO utilized to optimize the model towards short-to-long preferences using DPO-style objectives upon long contexts. Furthermore, to maintain the short-context performance, we incorporate short-to- long constraint in LongPO by applying Kullback-Leibler (KL) divergence between the response distributions to short and long inputs, respectively. This constraint, inspired by the KL constraint in RLHF (Ouyang al.) 2022; Stiennon et al. 2020), guides the policy model minimize the deviation from its short-contex output distribution when giving the long context during training. We found that this straightforward constraint largely enhances the retention of short-context performance after the long-context alignment.
We apply LongPO -istral-7B-Instruut--ie (Jiagg et al. 2023) and Qwen2.5-77 5-7B-Instruct while iteratively extending their context lengths up to 512K, with the self-generated short-to-long pref- erence data only. The experimental results demonstrate that LongPO, as long-context alignment method, surpasses naive SFT and DPO by large margins (over 10 points) in both long- and short- context tasks. Notably, LongPO fully retains the performance of short-context LLMs after long- context alignment, whereas SFT and DPO yield substantial performance degradation (10-20 points on most tasks). terms long-contedt performance, LongPO largely improves the Mistral-7B- Instruct-v0.2 by 25.45 points coBench. Specifically, depicted in Figure the resulting model
2

---

Published as a conference paper at ICLR 2025
comparable with superior long-con long-context LLMs various scales (e.g.. Boanral- of 39.27 vs. GPT-4- 128K 34.81 on ooBench), despite the latter often involving extensive continual training on hundreds of billions of tokens (Dubey et al. 2024) labor-intensive long-context data annotation (Zeng et al., 2024). These findings underscore the efficacy of our proposed method in addressing the challenges of long-context alignment while simultaneously preserving short-context capabilities, offering more efficient and balanced approach to the development of long-context LLMs.
PRELIMINARIES
In this section, introduce two key methods for aligning language models with human preferences: Reinforcement Learning from Human Feedback (RLHE, $2.1 and Direct Preference Optimization (DPO, 2.2
2.1 RLHF
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022 Stiennon al. 2020)
aims to optimize the policy model we to maximize rewards while maintaining proximity to refer- ence policy Tet. Formally, the objective is
max
Do m (T (To(
BI ( ( (x) Teer(s/ x)),
where ra is the reward model that has been trained on ranked responses reflect human prefer- ence, 33 is a hyper-parameter controlling the deviation from reference policy, and Dxr. denotes the Kullback-Leii Leibler divergence. Typically, both ma and Teet are initialized with identical model.
2.2 DPO
Considering the instability and difficulty of RLHF training, DPO (Rafailov et al.) 2023) offers an alternative approach by reparameterizing the reward function that incorporates the optimal policy:
To x) xT)
r(z; v) Bllog
log 7(x),
Tea (y
preference data D, which consists of paired responses (Vw, y/) to an instruction x. Specifically, the Yo and I/ represent the preferred (winning) and dispreferred (losing) responses, respectively, based human preference. Inspired by the Bradley-Terry (BT) theory that models the preference distribution p" by
where 7(x:) is the partition function. DPO assumes
p" (yw V1 2) = (r(2, Vue r(.,)()))))))))))) y))),
(3)
where a is the sigmoid function. DPO derives the preference optimization objective for the policy model Fy as
- LDvo(To: neat) = ((a MK)D WDDD [o(ro(I, vw) - (2 v1))
=- log Blog ( ww |x) (oe x)) Teet x
log No (v) xx)
(4)
3 LONGPO: SHORT-TO-LONG PREFERENCE OPTIMIZATION
Motivated by the challenges of data annotation and performance balance during long-conted align- ment, we introduce the Short-to-Long Preference Optimization (LongPO), to effectively empowers a short-context L.M self-evolve to a long-contex, counterpart while preserving its original short- context capabilities. The foundation LongPO lies in the transfer of capabilities deeply ingrained during short-context alignment long-context scenarios by learning from short-to-long preference ($3.1). Additionally, LongPO incorporates a short-to-long constraint based on the KL divergence between short- and long-contess models during training, maintain the short-context performance in simple yet effective way ($3.2). In $3.3 we present the details curating short-to to-ooto-o prefer- ence data without external guidance and self-evolving long context training process using LongPO.

---

Published as a conference paper at ICLR 2025
3.1 LEARNING FROM SHORT-TO-LONE PREFERENCE
As outlined in $2 aligning LLMs with human preference typically relies on datasets comprising ranked responses identical prompts or instructions. However, in long-context scenarios, con- structing such datasets becomes impractical due to the extensive effort required for annotation. To circumvent the external data annotation, we leverage the short-to-long preference to internally trans- fer capabilities well rell-established in the short-context alignment of LLMs to long-context counterpart.
Concretely, we assume access solely to short-context LLM Ty that has been well aligned. Given a long imput =y [CL: [L] where CL is the long context and IL is the instruction, we can acquire the response yL ~ ( z1) by conditioning on the entire context. Due the limitations of iTs in handling long contexts, y.. is likely to be of lower quality.
We then hypothesize an ideal extractor that can rigorously identify and extract all essential infor- mation Cs within Ct relevant to addressing J1:
(5) By querying the instruction 4t based on Cs, we obtain a new answer yss ~ TT(y æs), where Is [Cs; As Cs is shortened context for I, the well-aligned short-context model TTs should be capable of producing a high-quality answer that aligns with human preferences.
Cs F(C,(C).
Intuitively, y/s serve as high-quality answer even when giving the whole long context, as its conditioned context is self-contained for instruction Jt Hence, we definite the short-to-long preference distribution SL based Bradley-Terry (BT) model following Eq. (3):
ST. (ys
|L) = a(r(1L, (s) (L))
(6)
We now steer a policy model Te (initialized with TT) to follow the preference distribution p"1, forming the LongPO objective:
Lunggo (To; T (ag. ).) [o<ro us) (m1. vL)),,
where DSL is the short-to-long preference data consisting of quadruples (25, Us. v1.) This objec- tive encourages the policy model to consistently accommodate the well-aligned short-context pref- erence while deviating the under-aligned long-context preference. Therefore, LongPO internally transfers preferences from short to long contexts without requiring external supervision, effectively addressing the challenge of long-context data annotation.
3.2 SHORT-TO-LONG CONSTRAINT
Long-context alignment often leads imbalance between long- and short-context performance. While this issue can be mitigated by carefully calibrating the scale and mixing proportion of long and short data across various context lengths, such an approach is resource-intensive and time- consuming, Moreover, an excessive incorporation of short-context data may inadvertently lead to insufficient long-conted alignment. In LongPO, we recognize that the degradation in short-context performance during long- conteottext alignment may be attributed to an improper (or missing) constraint current alignment methods.
Specifically, the RLHF and DPO objectives (implicitly) include a KL divergence term, 3KL (
a)) || Ty x)]. which serves as a constraint to prevent excessive deviation from the reference model in Eq. ① For long input Tt. this constraint C is expressed
C DKa [To( m)) || Teery|] m.)]
(8)
However, the reference model is typically the short-context model Fy itself, which is not adept at handling long contexts. This results in a problematic reference distribution Ter(y xy.), leading to undesired deviation from the short-context model distribution.
To address this issue, we propose a short-to-long constraint leveraging the quadruples introduced in Eq. (7). Recall that as contains all the essential information from Iı required generate satisfactory response, can serve as a proficient reference model conditioned on zs. While for an ideal reference model n capable of handling context lengths from short long, we should have:
DxI. (is [
mm.) || mer
[ Tex mier(y/ (ss) || ( sss]
4

---

Published as a conference paper at ICLR 2025
Instruciion Gencrotion ove Shart Contost
Conted
Gunoration
Response Generation
Nany Fotter and the Polpoopher's Shone io fontaey novel written by the Brft sh suthor K. Rowling...
content generate comprchensive questions..
Response
The book first published in the Uniiad Kingdom 26 Jane 1997 by Bloombbury. was published the United States fotlowing
Anewor Wirlk tan yeur of
Stone schieved widtorradd prpularit cunneriall ventoos lase Nowr ruek
Scholastic Corporation under tite harry Potter and the Sorcare's Stare. It most Britiin book anartss and other awards the US The book necched the the New York Times inst best-satting dettion August 1999.
Long Context
Shart toort-to-iaa Long Preferance
o derwee Resnctin the que infornatioon co clouid auiiion derpind poputatyy
hh buan translated into leeat 73 other languageo and matee farturu langth Film of same
Pejected Responne
Instruction: Hove popsiarr the Harry Potter and the Philoopherr' Stone years of publicatio??
Figure 2: The procedure of generating short-to-long preference data from step 1 to 7.
namely (y .i.) and Ts(y | Jrs) are identical distribution following Gibbs' inequality, We hence derive an adjusted short-to-long constraint between short-context reference model and "long- context" policy model given contexts of different lengths:
C' 3I. [1 ar.) || us(y) ms)]
(10)
This refined constraint ensures that the policy model Ta operating on long contexts does not deviate significantly from the short-context model मş when provided with the essential information. By enforcing this constraint, we aim to preserve the short-context performance during long-context alignment, thereby addressing the imbalance issue in a more principled manner.
By incorporating the short-to-long constraint in Eq. (2) we have refined reward function for long input æ1, (following derivation in Appx. $A.1
LongPO (11.) v) 3 Blog To xL)
+ 3 log Z (xI,,
(11)
(y) xs)
where æy is extracted from Iy as illustrated in Eq. (5). Hence we access the LongPO objective:
Longro (To:Ts) = - -E 12. (r (IL, LongPO (XL,
= -B logo Blog T xL) - -Blog To yL xL))
(12)
IT(ss xs) xs(m.. xs))
3.3 SELF-EVOLVING
Initialization. LongPO relies solely access to well-aligned short-contexi LLM, i.c., Ts. in conjunction with a long-context plain corpus. Note that the long-context corpus need not be metic- ulously crafted, it can be sampled and extracted from existing pretraining corpora of LLMs.
Construction of short-to-long preference data. The construction of short-to-long preference data DSL introduced in $3.1 assumes impractical extractor capable of retrieving essential in- formation from long contexts for each instruction. To satisfy this hypothesis, we reversely prompt vs generate instructions for shortened chunks within long documents. This ensures that the short context information is self-contained for instructions. Concretely, our data construction process involves two steps as displayed Figure 2
1. Instruction Generation. For each long document C2, we randomly sample a shortened chunk Cs and prompt the my generate an instruction via the Self-Instruct (Wang al.). To ensure the diversity of instructions, the model is prompted to generate an instruction pool first and then we randomly sample an instruction Jr from this pool.
2. Response Generation. Using the generated instruction 4, prompt πs to produce two responses: a chosen response y/S ~ Ts (y) æs) based on the short context as, and a rejected response yr. n (y m1.) derived from the long context .T.-
5

---

Published as a conference paper at ICLR 2025
Iterative self-evolving training with LongPO. LongPO employs iterative process extend LLM context length. Initially, a short-context LLM иy generates short-to-long preference data for documents of length L'. The resulting model after LongPO training, now capable of handling L' contexts, then serves as the new "short-context L.M"" for the next iteration, generating data for an extended length T.2 This process repeats, progressively increasing context length capacity.
As there are multiple short chunks Cs = (Cg)m= within a long document Ct. we collect the instruction-response triples (Ii vš. vi.) for each chunk within identical long document, to form a multi-turn dataset DSL We then aggregate the probabilities across all turns to produce multi-turn LongPO objective:
E To Cus xt) D Ts CU'S CT$)
E To (v at)) E IT Cot (3))
CMT Longro (T0
TT) -B (os on. M.
log a
Bogg
3 log
(13)
where I3 {[C}: Ii.):..: ([C,; Us {1r) and yv (. LLMs trained with LongPO do not necessarily involve continual training before, which may lead to instability when processing long contexts. To address this issue and stabilize the training process, we incorporate a continual training objective following Pang et al. (2024b). Specifically, we add the negative log- likelihood (NLL) loss over entire long chosen sequences SL= [aL: (41: v LongPO objective. Thus, our final training objective is:
(To: (T) + (To: = A CMT ( (T) + no (Si.)
Cg = A CMT Longro
(14)
[SL]]
EXPERIMENTAL SETUP
4.1
TRAINING SETUP
Data Curation Details. We curate the short-to-long preference data based on a long-context cor- pus sampled from the Book and ArXiv subsets of Long- De--aaaaCollettio/// and the GitHub subset of RedPajama (Computer) 2023). For a specific target length (e.g.. 128K tokens), we filter the corpus to include only documents that shorter than this length but longer than 64K tokens. Each long document is then segmented into chunks of up to 32K tokens, with a maximum of 4 randomly- sampled chunks retained per document. For instruction generation, we prompt short-context models to generate 4 instructions per document, from which we randomly select one for further use. Af- ter filtering undesired (censored and repetitive) responses, we collect 45K, 16K, 2.5K multi-turn instruction-response samples of 128K, 256K, and 512K tokens for Mistral-7B, and 32K samples of 128K tokens for Qwen2.5-7B. More details are listed in Appx. $B.1
Training Details. We the context length of Mistral-7B and Qwen2.5-7B using our LongPO on short-to-long preference data specifically generated by models themselves. The training pro- cess begins with utilizing Mistral-7B//Qwen255577 to generate data with a length of 128K and ex- tend the context length to 128K. To investigate the scalability, we utilize the resulting Mistral-7B- LongPO-128K to generate data with lengths of 256K/512K and further extend the context length to 256K/512K. We leverage Deepspeed- Ulyduu-lssssss (Jacobs et al. 2023) for sequence parallelism and employ Flash Attention (Dao al.) 2022 Dao 2023) for efficient computation. All models optimized using the Adam optimizer (Kingma & Ba 2015) with a learning rate of 5e-7. We set the
margin in Eq. 13 to 0.1 and the weighting factor in Eq. (14) to 0.01. RoPE 0 is set as le7 and
the batch size is set as 8. More details are listed in Appx. $B.3
4.2 EVALUATION BENCHMARKS
We assess both the long- and short-context capabilities of our models against baselines. The long- context evaluation utilizes the following benchmarks:
Bench Zhang et al.). We evaluate all models on three tasks in this benchmark: sum- marization (En.Sum), long-book question answering (En.QA). and multi-choice question- answering (En.MC). The evaluation length is beyond 100K.

---

Published as a conference paper at ICLR 2025
RULER (Hsieh al. 2024 This benchmark comprises four types of synthetic tasks across variable sequence lengths (4K to 128K): Needle-in-a--baysacck (NIAH) retrieval, Multi-hop Tracing with Variable Tracking (VT), Aggregation, and Question Answering (QA). We exclude the Aggregation tasks, which involve word frequency counting within the context, since they present challenges in word counting beyond mere long-context pabilities that current LLMs still struggle in.
LongBench-Chat (Bai et al. 2024). This benchmark assesses Instruction-following abil- ities over long contexts (1OK to T00K tokens), employing GPT-4-128K as an impartial judge to evaluate model-generated responses. We filter out the English samples for fair comparison across different models.
For short-context evaluation, we employ MMLU (Hendrycks al., 2021). ARC-C (Clark et al,
2018), Hellaswag (Zellers et al.) (019) and Winogrande (Sakaguchi et al., 2019) for assessing the general language understanding and reasoning capabilities, and MT-Bench (Zheng et al. 2023 for assessing instruction-following capability. More details are listed in Appx. B.2
4.3 BASELINES
We train our LongPO on Bestral-BB-Instruceee (denoted as Mistral-7B) and Qwen2.5-7B- Instruct (denoted as Qwen2.5-7B), comparing them against range of powerful LLMs including GPT-4-12KK, 1eee2--7BBInsrruue (Yang et al.) 2024). LLaMA-3.
LLaMA-3. 1-BBB, GLM- 4-9B-Chat, GLM-4-9B Chaa- Twxt--eexhChatt IM (Liu et al. 2024b), and farn-Mistral-7b- 128k (Peng et al. 2023). Additionally, we establish baselines using Mistral-7B trained with conven- tional SFT and DPO on the same dataset used by LongPO.
For short-context evaluation, we primarily compare the performance of naive LLMs against their counterparts post-trained with SFT, DPO, and LongPO on our synthetic data. To provide a more comprehensive comparison, we also include two series of open-source long-context language mod- els: GLM-4-9B- Chat versus GLM-4-99 L.MM---9-Chht-IM and LWM-Text- Chat-1M. This allows to assess the effectiveness of our LongPO to maintain the short-context performance during long-context alignment, comparing with baselines utilizing various strategies.
128k versus LWM-Texx
RESULTS AND ANALYSES
In this section, demonstrate the exceptional effectiveness of LongPO through two types of com- parisons: (1) comparison with naive SFT and DPO trained on identical models and datasets; (2) comparison with SOTA long-context LLMs.
5.1 COMPARISON WITH SFT AND DPO
We first compare LongPO with conventional SFT and DPO using identical LLM (Mistral-7B). All models are trained on equivalent self-generated datasets, as detailed in $4.1 Given the inability of SFT to leverage preference data, we apply to the instructions paired with chosen responses.
LongPO exhibits superior performance over SFT and DPO. The experimental results, illus- trated in Table I] reveal consistent and substantial performance gains (10 to 20+ points) of LongPO over SFT and DPO across diverse range of long-context tasks. Crucially, as depicted in Figure 3] LongPO maintains robust short-context performance compared with original short-context LLMs (59.99 vs 59.15 on MMLU), whereas SET and DPO exhibit notable degradation in short-context scenarios after long-context alignment process.
The performance disparity between LongPO and SFT can be attributed the explicit integration of short-to-long preference in LongPO, which is either absent or merely implicit in the chosen re- sponses utilized by SFT. While both LongPO and DPO leverage the proposed short-to-long prefer- ence data, the pivotal difference lies in the short-to-long constraint introduced in $3.2 The marked performance gaps between LongPO and DPO, observed across both long- and short-context tasks, highlight the effectiveness of the proposed constraint for successfully mitigating the problematic limitations in DPO and retaining the short-context performance during long-conten training. More ablations detailed in $5.3

---

Published as a conference paper at ICLR 2025
Table 1: Long-Contess Performance of our LongPO compared with baselines. Higher is better for all metrics. Results marked with 5 are evaluated by ourselves, while other results of baselines are sourced from the original benchmarks. Full results on RULER are listed Table
Model
Train/Claimeu
oo Bench
RULER
LongBench-
Length
En.Sum
En. Bn.QA
En.MC
AVG.
NIAH
VT
QA
AVG.
Chat (EN)
GPT-4-12KK
128K
14.73
22.44
67.25
34.81
95.4
99.9
70.3
88.53
8.40
Qwen2-72B
128K
24.32'
7.03
72.05
34.47
88.6
95.7
66.7
83.67
7.72
I.LaMA 3.1-70B
128K
33.55'
36.08
69.00'
46.21
96.1
93.2
67.8
85.7
6.67
LLaMA 3.1-8B
128K
28.06
30.47
58.08
38.87
97.93
91.4
64.7
84.68
6,22
GLM-4-9B
128K
14.84
9.51
67.25)
30.53
96.51*
97.3*
64.8%
86.20
5.67
GLM-4-9B-IM
1M
28.3
9.7
68.6
35.53
98.2
99.4
69.4
89.0
5,03
LWM-7B-IM
4.33
0.0
3.06'
2,46'
87.20
57.5
56.4
67.03
1,25
128K
9.09
9.55
27.95
15.53
63.4
36.1
25.9
41.8
Mistral-7B
32K
22.13
4.93
14.41
13.82
72.60
74.40
$2.2
66.4
SFF
128K
23.44
13.45
53.21
30.03
88.73
79.64
51.08
73.15
4.25
DPO
128K
15.21
10.34
48.14
25.56
72.36
50.24
65.62
LongPO (iterl)
128K
27.05
23.51
67.25
39.27
96.88
96.49
64.81
86.06
5.42
LongPO (iter2)
256K
28.16
24.43
66.35
39.65
96.80
97.0
64.87
86.22
5.48
LongPO (iter3)
512K
29.10
27.85
66.67
41,21
97.28
97. 17.48
64.92
86.56
5.80
Qwen2.5-7B
128K
22.89
6.08
52.4
27.12
82.1
80.09
54.30
16.16
5.80
LongPO (iterl)
128K
32.06
17.32
72.05
40.48
95.81
89.71 1
59.4
81.64
5.75
Lo fiter!)
LongPO fiter2)
SFT
DPO
GLM
LWM
1.0
1/0
A16
Margin
-021
+1.3
-100
071
-177-4477
20.74
MMLU
ARC-C
Hellaswag
Winegrande
MT- WT-Bench
Figure 3: The margins of the short-context performance of Distral-7B-Longeee and baselines rela- tive to corresponding base model. GLM and LWM refer to the margins of GLM-9B- MM and LWM- 7B-IM over GLM-9B- MMMM----288- and LWM-7B-I28K respectively. MT-Bench metrics (E[0, 10]) are linearly scaled to [0. 100] for better comparability across tasks. See numerical results in Table 3
5.2 COMPARISON WITH SOTA LONG-CONTEX. LLMS
To further substantiate the efficacy of LongPO, we conducted an extensive comparison between our LongPO-traine Mistral-7B and leading long-context LLMs across varying model scales.
LongPO demonstrates exceptional competitiveness at similar scale. As detailed in Table 1
LongPO demonstrates formidable competitiveness in terms of models at similar scales. For exam- ple, Mistral--BB-LonPPO significantly outperforms some established long-context models, including LWM-7B and YaRN-Mistral, across all long-contex: tasks in co Benchhh and RULER. Remarkably, Distral-BB-LongPO-- 128K surpasses GLM-4-9B (39.27 vs. 30.53 on ooBench and 86.06 vs. 86.20 on RULER), although the latter is training on manually annotated long context data spanning up 128K sequence length. Moreover, GL.M--99--IM an extension of GLM---9B trained on con- texts up to 1M tokens, demonstrates slightly superior performance than LongPO on the RULER benchmark. However, these performance gains come at the costs degenerated short-context per- formance (0.41 MMLU) and long-context instrrctioon-folloiingg capability (0.64 on LongBench- Chat (EN)) as illustrated in Figure 3) Notably, our models still outperform GLM-4-9B. 1M on oo Bench even trained with substantially shorter sequences. These results underscore the exceptional efficiency of LongPO in transferring performance from short to long contexts through self self-eevouuion,, thereby circumventing the need for extensive manual annotation.
8

---

Published as a conference paper at ICLR 2025
LongPO
LongPO SFT-Chbosn
SFT-Choss
80
RULER-NAA
50
20
1000
3000
5000
1000
5000
5000
Training Steps
Training Steps
Figure 4: Long- and short-context performance comparison among LongPO, SFT on chosen re- sponses (SFT-Chosen), SFT rejected responses (SFT-Rejected), DPO, and SFT chosen sponses with short-to-long constraint SSFT-Chose-CConstrinn)
Long-context annotation is not sufficient. The superiority of our approach is particularly evident in the En.QA task within coBench, which involves complex free-form question answering over extensive book-length contexts. In this challenging task. our models surpass both GLM-4-9B and GLM-4-9B-IMM by substantial margins (10+ points). The inherent difficulty of such task, which poses challenges even for human annotators, highlights the limitations of relying solely on manually annotated long-context data. By effectively transferring short-context capabilities to long-context scenarios, LongPO demonstrates superior scalability and efficacy across diverse and intricate tasks.
Superior LLMs Yet Dominate Long-Context Scenarios When benchmarked against leading models such as GPT-4-128K, our LongPO-trained models still exhibit comparable or even superior long-context performance (e.g., Mistral-7B-Lan -Boroll---- of 39.27 vs. GPT-4-128K of 34.81 on oo Bench), despite being based on significantly smaller Mistral-7B. This observation reveals that even the most advanced LLMs have not yet achieved the same level dominance in long-context scenarios they have in short-context tasks. This performance gap can be attributed primarily the scarcity of high-quality, large-scale long-contes training data. The dearth of such data is particularly impactful for larger LLMs, given the established scaling laws in language model training. This finding underscores the potential of LongPO for enhanced performance without the requirement for externally annotated long-context datasets.
5.3 ABLATION STUDIES
We conduct comprehensive ablation studies to investigate the efficacy of components in LongPO:
Effectiveness of short-to-long preference. The core of LongPO is learning the short-to-long pref- erence between chosen and rejected responses given short and long contexts, respectively. To evalu- ate this component's 's effectiveness, we compare LongPO with two baseline methods: SFT on chosen responses (SFT-Chosen) and on rejected responses (SFT-Rejected), SFT-Chosen implicitly incor- porates short-context preference, while SFT-Rejected entirely omits it. As illustrated in Figure 4] LongPO consistently outperforms both SET variants in long-context performance (RULER-NIAH) throughout the training process. This substantial improvement underscores the efficacy of our short- to-long preference approach in enhancing long-context capabilities.
Effectiveness of short-to-long constraint. To assess the impact of our short-to-long constraint, compare LongPO with DPO upon short-to-long preference that removes this constraint. As evi- dent in Figurr 4 the unconstrained DPO demonstrates markedly inferior performance throughout the training process, both in long- and short-context tasks. Notably, short-context capabilities degrade rapidly in DPO during the initial training. Conversely, when we apply our short-to-long constraint to naive SFT without explicit short-to-long preference, the model maintains short-context performance on par with the original LLMs, even after long-context alignment. These results demonstrate the crucial role of short-to-long constraint in preserving short-context capabilities.
Impact of NLL loss. We investigate the effect of incorporating negative likeliiildood (NLL) loss over long context input and chosen response in Eq. (I4) during LongPO training. As shown in Figure 4 removing the NLL loss significantly degrades the long-context performance of LongPO

---

Published as a conference paper at ICLR 2025
across the training procedure. Specifically, the convergence of training for long-context performance becomes slower. This demonstrates the crucial role NLL loss in enhancing long-contens capabil- ities without resorting to continual training on long data.
RELATED WORK
Alignment of LLMs. Aligning Large Language Models (LLMs) with human preferences and val- ues has been crucial to unlocking their full potential from large-scale pretraining. The typical align- ment process begins with Supervised Finetuning (SFT) on annotated instruction-ressonse pairs. This is followed by Reinforcement Learning from Human Feedback (RLHF), which aligns LLMs more closely with human intentions through reward model training and policy optimization (Christiano) 2017 Ouyang al. 2022 Bai et al. 2022 Stiennon et al. 2020). To streamline RLHF train-
et al.
ing. Direct Preference Optimization (DPO) (Rafailov et al. 2023)) and its variants (Fthayyrajj et al.)
Azar al. 2023 Pang al.) 2024a Hong et al. 2024 Meng al. (024) have been pro- posed, eliminating the need for explicit reward model training by learning preferences directly from human-ranked response pairs. While these alignment methods have shown significant success, they heavily rely on human-annotated annotated data. This reliance becomes problematic for long-context data,
2024
where human annotation is both challenging and potentially less reliable.
Long-context extending of LLMs. Extending the context length of LLMs has been approached through various methods. Some techniques involve scaling the rotary position embedding (Su et al., 2022] followed by continual training on a small corpus of long documents (Chen al.) 2023b] Peng et al.. [023 Rozière et al. [023]] Chen et al. (223a)). Alternative approaches, such as those proposed by Jin et al. (2024): An et al. (2024), introduce hierarchical or chunked attention mechanisms to ex- tend context length without additional training. However, these methods often involve limitations in practical applications. Recent advancements include the work of Dubey et al. (2024), who proposed massive long-context corpus (800B tokens) and incorporating small fraction (0.1%) of long-context data during SFT to enhance long-context capabilities. Zeng et al. (2024) utilizes human-annotated long-context data for SFT and DPO to align long-context LLMs. Despite their effectiveness, these methods require either extensive training or human annotation of long-context data, making them prohibitively expensive and lack scalability.
continual pretraining
Self-Evolving LLMs. Recent works (Yuan al. 2024: Liu et al. 2024a Li et al. 2024) have
unveiled the remarkable capability of Large Language Models (LLMs) to evolve from relatively weak to significantly stronger performance through self-augmented data. Yuan al. (0024;;
Liu et al.] (2024a) leverage iterative training on model-generated responses, ranked by LLM-as- a-Judge (Zheng al., 2023) prompting, to enhance model itself. Li et al. (2024) introduces the instruction backtrans to produce self-augmenting data that further enhances model capabili-
ties. Our work first extends the self-evolution property to the context length, develop long-context LLMs without relying on external annotations.
7 CONCLUSION AND DISCUSSION
In this work, propose LongPO, novel long-context alignment method that enables LLMs to ef- fectively transfer their short-context capabilities to long-context scenarios. Our approach addresses key challenges long-context alignment by leveraging intrinsic model knowledge, eliminating the need for external long-context annotated data. LongPO is built on short-to-long preference data, comprising paired responses for the same instruction given a long context and relevant shortened chunk, respectively. By steering the policy model to learn from the discrepancies within these paired responses, LongPO facilitates the transfer of established capabilities from short to long contexts. In addition, LongPO incorporates a short-to-long constraint using KI divergence, that effectively pre- serve short-context performance during training. Experimental results demonstrate that LongPO significantly improves long-context performance across various tasks, outperforming existing align- ment methods and even surpassing more sophisticated models. Importantly, this improvement is achieved without sacrificing short-context proficiency. The success of LongPO highlights the poten- tial of leveraging internal model knowledge for alignment tasks, opening new avenues for efficient adaptation of L.LMs to diverse context lengths.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by DAMO Academy through DAMO Academy Research Intern Program.
REFERENCES
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. traeeing-free long-context scaling of large language models, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310. 12036, 2023. URL wtl.https:// /api semant cschol ar. aararro org/
Corpus Norpus1D:26428855
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hemanneez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawen Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield -ooddds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedhack. 2022. URL http::ttt//o... aariiiiiivvv or org.orggaaeoooo abs 22222.2222222222 08073
Yushi Bai, Xin Lv, Jiajie Zhang. Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. 2024. URL https://arxx wwt./asg/g//eee.eo.o /2010 18058
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard
(002--2224)
open 1m leaderboard, 2023.
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Li Bing. Clex: Continuous length extrapolation for large language models. ArXiv, abs/2310..14444 2023a. URL https://api.
semant cscho ar.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. 2023b. URL httts::/arxiv..org/
abs/ 2306 15595
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv, Abs//803.00557, 2018. URL https::/t////.... /abas://aple /api. www.::seaaaticcccccccccoec org CoxpusID:3922811
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023. URL http://///ggggie.ooooooooteeeeee
-Data
Tri Dao. FlashAtteemionn-2: Faster attention with better parallelism and work partitioning. 2023.
URL
Tri Dao, Daniel Y. Fu, Stefano Ermon. Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems, 2022. URL /tt
14135

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint DrXiv2247.217883 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv, abs/2402.01306 2024. URL https:// api. www.emmtniiiemmm csche ar org/ :D:rpusID:26740000
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL http:tttt//eeeeeoo
org/ wrggrr/crrs/////
12608602
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt, Measuring massive multitask language understanding. In International Confer- ence on Learning Represenlations, 2021. URL hhttps://oppnnsee wttp:e/wwewttrrra//// forum?id-
net
d7KBjmI 3GmQ.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. ArXiv, abs/2403.0760 2024. htllitttaas/ppp..eeanttccccc.
org/CorpusID. : 2262633333333
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0660 06654, 2024.
Sam Adé Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Ra- jbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. ArXiv, abs/2309.14509 2023. URL https:
//api
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, 0ls/2310.06825, 2023. URL https::/apg: sewanticschoteoo /ttE::://pp.mmmnnicccchoo... http:///oo..aaaiioooooooooooo//o 263830494
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend I1m context window without tuning, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Yoshua Bengio and Yann LeCun ). 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htRhttp: //arxiv.cmm //axxx.. org/ /1152./111.... 6980
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2024. URL httpps://arxiv:
org/abs/22 abs//308. 06259
Aiwei Liu. Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation, 2024a. URL httLtt::s///xxxiiii wttp:/xxwwixx.rr///x//ee/ 11907]
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024b. URL htttts://aaa//t www.eorgaaaao abs/
2402. 08268
Yu Meng, Mengzhou Xia, and Dangi Chen. Simpo: Simple preference optimization
with a reference-fre reward. ArXiv, abs/2405.14734, 2024. URL [htpps: //app.
semant i cscholas wwwhhhllr.rggg/..oo httpa//arr..ro//ooooaaaaotaaaaa.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.00155. 2022. URL http::::: //tps://app. /api http:///aap.ssmnnsisssss.... www.sssttcccccccolrrr......
CorpusID:24642880
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. ArXiv, abs/2404.19733, 2024a. URL
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason
Weston. Iterative reasoning preference optimization, 2024b. URL
Sbbs/404.19773
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole Yarn: Efficient context window extension of large language models.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. 2023. URL https://
btxiv.org/abs/// 2tt.//200081115100 12950
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint arXiv:1907.1064/ 2019.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 3021, 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu, Roformer: En- hanced transformer with rotary position embedding. 2022. URL 2104. 09864
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct Aligning language models with genf-genertted instructions. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume org/ 2023 acl long. 754
1; Long Papers). htllhttss//aclaannn
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. Interna- lional Conference Learning Representations, 2022. URL htttps://
net/
forum?id=gf (orum??dFFFFf
An Yang, Baosong Yang. Binyuan Hui. Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang. Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan. Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://aras org/ 2407 10671
13

---

Published as a conference paper at ICLR 2025
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbuatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXiv, abs/2401.10020, 2024. URL/tttss://aaa //api.
semanticscholar wwt.:oog///aaaee.e ID 6035299
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Annual Meeting of the Association for Computational Lin- guistics, 2019. URL https: api. semanticsccoo..............
org/ wwt.ooo///////eeee Cttp//orrsssiDi.. 159044722
Team Glm Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Ming yue Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiaoyu Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang. Xixuan Song, Xunkai Zhang, Yi An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhenyi Yang, Zhengxiao Du Zhen-Ping Hou, and Zihan Wang. Chatglm: A family of large language models from glm-1306 to glm-4 all tools. ArXiv, 122224266.2773, 2024. URL https: 7/api www.smemtttnnoooeee..c.occ wtt.p/ooaa.....oooorrroro.....2. :
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. Benchh Extending long context evaluation beyond 100K tokens. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long acl acl-long. 814
Papers). URL htLhttps://
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.005683 2023. URL
https: aapi. www.sttiinnn.ooo...... htt.:aa....rrroopoo..02222222220
14

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DERIVATIONS
A.1
DERIVING THE LONGPO OBJECTIVE
In this section, will derive the reward function of our LongPO objective in Eq. 11 by incorporat- ing short-to-long constraint in Eq. (IO). Starting from RLHF objective in Eq. ① with short-to-long constraint, have
max E 2y D.y
(15)
Following the DPO derivation process Rafailov et al.
2023). we have:
max (n) (-())))
(.L, 31lx1 (2)
|| [s(s)(
r(z..)) - log (2)ar)
max (s) (yl- By
as(y)ars)
* (v)xL)
min E
D Dea R. (()r)))
log Ts (1)(25s))
3 (222
(()(22L
min R (z) Du R (v)l)))
log
log (
(3(( exp (3r(Øl. v))
(s
(16)
where we have partition function:
Z(L Σ TT (y) exp (Z)(o..))
The partition function is only related to zt, Vy, and original short-context LLM Ts. Hence have the optimal solution following Rafailov et al. (2023):
1
Ts (vlars exp 1 (f)()))))
t (v|=L) (xL,,z))
(17)
The optimal reward function would be derived:
r* (XL, ) Blog nt' (y/ Blog I(LL,
(18)
ITS (g)
We thus have:
cxp (x* p² (yn yalo1) exp (IL. + exp (r" (IL,
cxp B log
x' (v]llt) + Blog Z 2xL)
us( (u))
exp
Blog (v)(aL) ( (x))
+ B 7 ( xs)) + exp (1 Blog (v(la)))) ))(
+ Blog 7 (xL,
(Blog ** (wulz)) B log ((((a))) (s(valus)
exp
(ville)))
T" (V2(=L) n² (2111_L) 3 log
Blog TT Ts(yilas) (zs)
By optimizing the my towards the optimal policy n², we finally access the objective of LongPO in Eq. 12
B EXPERIMENTAL DETAILS
B. DATA CONSTRUCTION DETAILS
We prompt the Mistral-7B 7 7B-IIntrruct-v022 to generate instructions with decode parameters of temper- ature T 0.7 and p 0.9. The prompt of Self-Instruct to generate instruction pool is shown
15

---

Published as a conference paper at ICLR 2025
Based on the content presenled above, generate 4 comprehensive English questions that test reader's comprchension, analytical skills, and ability to extract and interconnect key themes and ideas across the entire document.
Each question should:
Encourage the reader draw connections between different sections concepts within the text. only recall information but also synthesize and summarize material coherent manner.
Challenge the reader
unique focus, avoiding repetition ensuring broad coverage the document's content.
Stimulate critical thinking by requiring the application evaluation of text's information
broader contexts or hypothetical scenarios, relevant.
5. Ensure the question clear and unambiguous.
Please directly give the questions without verbose illustration, and format the questions numerically from "1:" to "4:""
Figure The prompt for generating instruction pool.
For generating the corresponding responses, directly concatenate the short long context with corresponding instructions and adopt the greedy decoding to maintain the deterministic behaviour of LLMs. As shown in Figure 6 the model would tend to prefer the high-quality chosen response and deviate from the low-quality rejected response over long context, hence improve the long-context capabilities.
Figure 5
B.2 EVALUATION DETAILS
On long-context benchmarks InfiniteBench and RULER, we evaluate our models and all baselines following the settings in the original benchmarks. For short-context evaluation, we utilize the Im- evaluaton -hanness-- framework (Gao et al. 2024) and following the evaluation settings in (Beeching) al. 2023); 5-shots for MMLU, 25-shots for ARC-C, 10-shots for Hellaswag, and 5-shots for Winogrande. We use GPT-4-Turbo- P0-Prr the judge for MT-Bench and LongBench-Chat evaluation.
B.3 MORE TRAINING DETAILS
Leveraging the DeepSpeed- Ulysses sequence parallel framework, we train the Mistral-77B/Qwen2.55
7B with sequence length of 128K an 8xA800 80GB, achieving throughput of 4,401 tokens per second. For sequence lengths of 256K and 512K. the models are trained on a 16xA800 80GB, yielding throughputs of 4,120 tokens per second and 2,744 tokens per second, respectively. To facilitate comparison with standard L.M alignment methods, we train Mistral-7B using SFT and DPO utilizing the same short-to-long preference data of LongPO. For DPO training, apply the same settings as LongPO outlined in $4.1 but excluding the short-to-long constraint of LongPO introduced in $3.2 Since SFT cannot utilize paired responses within preference data, train using only the chosen responses provided alongside long context inputs. The hyperparameters for SFT remain unchanged, except for increase in the learning rate to 2e-5.
16

---

Published as a conference paper at ICLR 2025
-400
2043
800
Therddng Sugn
Saging Stecs
The rewards for chosen response during training.
(b) The rewards for rejected response during training.
Figure 6: The chosen and rejected rewards during the training of Mistral-7B-LoonnO--1222
Table 2: Full results on 13 tasks of RULER benchmark. The bold values denote the average score of tasks RULER over various context lengths.
Model
Category
4k
8k
16k
32k
64k
128k
AVG
99.69
98.45
97.82
95.24
74.56
26.86
82.
VT
99.88
99.72
96.24
96.44
81.44
6.84
80.09
Qwen2.5-7B-Instruu
AGG
92.52
89.78
92.08
81.93
62.48
28.23
74.50
QA
71.00
65.30
64.00
58.70
46.80
19.99
54,30
AVG (13 tasks)
94.19
92.11
91.61
87.66
68.96
24.47
76.50
NIAH
99.64
98.97
97.80
95.54
94.80
88.15
95.82
VT
99.96
99.92
96.12
86.24
78.20
77.80
89.71
Qwen225-77H..onpP0-1288
AGG
95.50
86.12
91.75
82.56
66.31
49.81
78.67
QA
70.00
64.00
62.70
57.70
$3.00
49.00
AVG (13 tasks)
94.47
91.69
91.34
87.00
82.71
75.43
87.11
NIAH
99.43
98. 98.64
98.09
97.84
95.82
91.44
96.88
VT
99.40
99.16
98.08
96.36
92.80
93.12
96.49
Mistral77B-LongP--122k
AGG
88.31
82.91
92.23
72.775
46.305
46.79
71.55
QA
71.10
70.15
65.80
61.00
$4.20
64.81
AVG (13 tasks)
93.36
91.88
92.35
88.94
82.61
78.97
88.02
NIAH
99.16
97.79
98.02
97.76
96.53
91.54
96.80
VT
99.40
99.20
97.96
97.72
94.21
93.52
97.00
Mistral-7B Mistral7BB-LongP-22555 LonnPo-2nP6K
AGG
87.40
76.59
89.03
72.20
45.17
44.47
69.14
QA
71.50
69.50
66.70
64.30
60.80
56.40
64.87
AVG (13 tasks)
93.11
90.28
91.81
88.68
82.95
79.04
87.65
NIAH
99.19
97.78
98.06
97.69
96.62
94.36
97.28
VT
99.44
99.16
98.04
97.80
95.92
94.52
97.48
Mistral-7B- 71---ooooP-51111
AGG
87.56
76.71
72.70
44.93
69.22
QA
71.40
69.50
66.40
64.50
60.60
57.10
64.92
AVG (13 tasks)
93.14
90.29
91.78
88.75
83.07
80.97
Table 3: Performance short-context tasks.
Model
MMLU
ARC-C
Hellaswag
Winogrande
MT-Bench
Mistral-7B-IInstruct-vl
59.15
59.26
83.2
78.4
6.34
59.99
59.34
82.99
78.53
6.35
Distral-7B-LongPO-222
59.47
60.28
83.14
78.14
6.38
59.51
60.58
82.87
77.66
6.34
74.28
67.15
81.41
74.66
7.30
Qwen2.5-7B--LoopP0--123
73.64
65.70
80.82
74.98
7.62
17


Reviews:
[{"review_id": "oNJAv9VYw1", "reviewer": "Reviewer_FWUE", "summary": "The paper proposes LongPO, a novel approach for extending LLMs' context length through self-evolution and preference optimization. The method leverages the model's existing short-context capabilities to generate preference pairs, using responses from short and long contexts as chosen/rejected samples. The approach incorporates a short-to-long KL constraint to maintain short-context performance while improving long-context handling. The authors demonstrate improvements over baseline Mistral-7B-Instruct, achieving competitive performance with larger models like GPT-4 on long-context tasks.", "questions": "1. How does LongPO compare with recent methods like LongLoRA in terms of computational efficiency and memory requirements?\n\n2. Could the authors provide comparisons with fine-tuning-free approaches and analyze the trade-offs?\n\n3. Have the authors explored combining LongPO with existing position embedding extension methods?\n\n4. What is the computational overhead during inference compared to other long-context adaptation methods?\n\n5. How does the method scale with different model sizes and architectures?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Novel Framework:**\n   - Introduces an innovative self-evolution approach for long-context adaptation\n   - Presents a theoretically sound extension of DPO for long-context scenarios\n   - Successfully maintains short-context performance during adaptation\n\n2. **Technical Merit:**\n   - Well-justified short-to-long KL constraint\n   - Clear theoretical foundations and derivations\n   - Effective integration of NLL loss for training stability\n\n3. **Results:**\n   - Achieves competitive performance with larger models\n   - Demonstrates effective preservation of short-context capabilities\n   - Shows promising results without external annotation", "weaknesses": "1. **Insufficient Comparisons:**\n   - Lacks direct comparisons with recent long-context methods \n    (e.g., \"Effective Long-Context Scaling of Foundation Models\", \"LongLoRA\")\n   - No analysis against parameter-efficient approaches\n   - Missing comparison with fine-tuning-free methods \n    (e.g., \"LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning\")\n\n2. **Limited Analysis:**\n   - Insufficient analysis of computational costs and training efficiency\n   - No investigation of potential combinations with existing methods\n   - Limited exploration of different model architectures and scales\n\n3. **Practical Considerations:**\n   - Missing discussion of memory requirements\n   - Limited analysis of inference-time computational overhead\n   - No investigation of scaling behavior with different model sizes\n\n\n**Suggestions for Improvement**\n1. Include comprehensive comparisons with recent long-context adaptation methods\n2. Add analysis of computational requirements and scaling behavior\n3. Investigate potential combinations with existing approaches\n4. Provide more detailed ablation studies\n5. Include practical considerations for deployment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIlz6bsJ3K", "reviewer": "Reviewer_rbSY", "summary": "This paper introduces a novel training method, LongPO, which can transfer the alignment capability on short-context to long-context scenarios.\nBy guiding the model to self-generate the training data and applying a DPO-based training objective, LongPO can greatly improve the model's performance on long-context tasks.\nExperiments on the Mistral-7B model show promising results for the LongPO method.", "questions": "1. Why not utilize the open-source training data as the source data for generating the training data in your paper? I noticed that the authors utilize the Instruction Generation method to ensure the diversity of instructions. However, based on my experiments, the instructions generated from the model have a relatively low quality (especially from the Mistral-7B-Instruct-v0.2 model) compared to the open-source long-form instruction data, e.g., LongAlpaca, namespace-Pt data[1].\n\n2. The experiments are mainly conducted on Mistral-7B-Instruct-v0.2, which already has a 32K context length. Why not conduct experiments on the short-context model (4K) like Llama-2 and the long-context model (200K) like Yi-200K?\n\n\n[1] Here I provide the link for namespace-Pt data, which I think has a relatively high quality: https://huggingface.co/datasets/namespace-Pt/projects/resolve/main/long-llm.tar.gz", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The direction of this paper is correct; long-context alignment is essential for long-context models nowadays.\n\nThe method is simple yet efficient; LongPO transfers the model capability from short-context to long-context scenarios by utilizing the KL term.", "weaknesses": "**Method**:\nI think the road of the overall approach is right, but the details should be improved. The method in this paper should be applied to the long-context models directly rather than doing both the context scaling and long-context alignment. The iterative self-evolving training is costly and complex, and it's better to directly use the positional extrapolation method (e.g., increase the value of RoPE base and conduct long-context post-training) to extend the context length and then utilize LongPO training to perform the long-context alignment.\n\n**Experiment**: Lack of details for reproducing the experiment. For instance, what are the settings of SFT and DPO for Mistral in Table 1? How much data do you utilize? How long does the training take? There are many settings and details that should be reported since DPO-based methods are often difficult to implement.\n\n**Writing**:\nThe derivation of RL formulas in the main body makes it difficult to understand the core idea the paper aims to present. I believe that if some of the derivations are included in the Appendix and only key formulas are retained in the main body, the readability will increase.\n\n**Figure**:\nThere are too many marked arrows in Figure 2 without corresponding textual explanations. To be honest, this figure gives me a rough idea of what the author is doing, but the specific details are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAB3MqTnB6", "reviewer": "Reviewer_6PbM", "summary": "This paper primarily talks about ways to reinforce the long-context capabilities of LLMs and proposes a novel method, called LongPO, that enables LLMs, which is solely good at short-context tasks in the beginning, to excel on long-context tasks by learning from self-generated short-to-long preference data. LongPO demonstrates promising long-context alignment performance, while retaining the effectiveness of dealing with short-context tasks. Furthermore, this work shows that large-scale human annotation for extended contexts is no longer necessary to excel the model’s long-context capabilities, which sheds light on doing long-context alignment solely with synthetic data.", "questions": "Do you think some other ways of constructing preference data using simple and clever rules/metrics, eliminating the need for human annotation but ensuring one of chosen or rejected data has a long context, are able to achieve similar alignment performance as LongPO? For example, thinking of constructing a new preference dataset by sampling some short-context data from the chosen group of existing preference datasets as the new chosen data, and sampling other long-context data from the rejected group of existing preference datasets as the new rejected data.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of constructing short-to-long preference data is simple, straightforward, and easy to implement.\n- The performance achieved by the proposed method is impressive compared to the baselines used in the paper, even surpassing some well-known proprietary LLMs.\n- The proposed method, LongPO, is able to retain the short-context performance by adding a KL constraint term to the objective, which explicitly guides the model to minimize the deviation from its short-context output distribution.\n- The paper is well written with related works being presented thoroughly to highlight its significant contribution.", "weaknesses": "- Though the construction process of preference data is pretty clear to me, I’m not fully convinced by the motivation that learning from these paired preference data, in which the longer one is dispreferred, enables the model to excel on long-context capabilities. And the authors haven’t provided good intuition or desired empirical evidence showing why doing preference optimization, where the dispreferred data contains the desired properties, is better than proper baselines, e.g., SFT with high-quality long-context data (in the paper, the SFT baseline is using either chosen short-context data or long-context data, in which the instruction is generated by short-context model).\n- The paper doesn’t discuss the bottleneck/limitation of the proposed method. Is LongPO able to scale to much longer contexts, such as 512k, 1 million? When does this method plateaus the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)