SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
Authors: Huashan Sun
Shengyi Liao
Yansen Han
Yu Bai
Yang Gao
Cheng Fu
Weizhou Shen
Fanqi Wan
Ming Yan
Ji Zhang
Fei Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Long Context Alignment
Large Language Models
Preference Optimization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A well-designed long-context preference optimization (PO) framework that improves both the effectiveness of original PO algorithms and the efficiency of data construction and training procedure.

Abstract: Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives. To address these limitations, we propose a framework named **S**h**o**rt-to-**Lo**ng **P**reference **O**ptimization (**SoLoPO**), decoupling long-context preference optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evidence. Specifically, short-context PO leverages preference pairs sampled from short contexts to enhance the model's contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency for the responses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model's ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes. Experimental results show that SoLoPO enhances all these algorithms with respect to stronger length and domain generalization abilities across various long-context benchmarks, while achieving notable improvements in both computational and memory efficiency.

Published as a conference paper at ICLR 2026
SoLoPO: UNLOCKING LONG-CONTER CAPABILITIES IN LLMs VIA SHORT-TO-LONG PREFERENCE OPTI-
MIZATION
Huashan Sun
Shengyi Liao Yansen Han
Yu Bai
Yang Gao
Cheng Fu'
Weizhou Shen' Fanqi Wan'
Ming Yan
Zhang'
Fei Huang'
Tongyi Lab, Alibaba Group
(liaoshengyí 1sy, yml www11666eeeaiaabbaaaaaaaaaaaaaaa
hanyansen@gmail. com
{hssum, yubai gyang) eew.bbittedu....
ABSTRACT
Despite advances pretraining with extended context sizes, large language mod- els (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimiza- tion objectives. To address these limitations, we propose a framework named Short-to-Long Preference Optimization (SoLoPO), decoupling long-context pref- erence optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evi- dence. Specifically. short-context PO leverages preference pairs sampled from short contexts to enhance the model's contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency for the re- sponses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model's ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes. Experimental results show that SoLoPO enhances all these algorithms with respect to stronger length and domain general- ization abilities across various long-context benchmarks, while achieving notable improvements in both computational and memory efficiency
INTRODUCTION
Long-text modeling is a cornerstone capability of large language models (LLMs) 776 17 77 38
While the input context size of LLMs has increased dramatically 44 [3], studies show that they can effectively utilize only 10-20% of this capacity primarily due to insufficient long- congexttext leaving their potential in long- onn-coonext scenarios largely untapped.
alignment [36
65
To address this issue, data augmentation methods [43 80
92 75 leverage advanced LLMs to generate long-dependency instruction-follie following data for supervised fine-tuning (SFT) and preference optimization (PO). However, as text length increases, these methods suffer from declining relia- bility and efficiency. Moreover, directly applying short-context training strategies to long-context scenarios may overlook the inherent discrepancies between the two settings, yielding suboptimal performance 15 37]. A different approach improves long-text alignment via training objective optimization. Fang al. [18] propose LongCE, which identifies tokens critical for long-text modeling and assign them higher loss weights during SFT. However, this approach incurs extra computation due to multiple forward passes to identify salient tokens. LongPO [IT] leverages responses generated with short contexts as positive examples in long-context direct preference optimization (DPO) [54]. Additionally, short-to-long constraint is introduced, which optimizes the DPO objective by replacing
Equal contribution.
1 Corresponding authors
"oode and data resources available at httpa/ach.tttpeegg.tooo/oo/////a

---

Published as a conference paper at ICLR 2026
Cpo BepDaeann BinDD Ve' a1 py
[ (n- (rots, Boo (x) 1)) ->>]
Run Time
Max Trainable Length
100
long-comlexst preferenco optimsization
75
2.1x
2.50
AautaPO
= Ee-DD snorr 3 Dr y/e i Winee (alhor))))
[s (1 [To(z, Bh) (, vi) m])]
120
min
122
9K
min
abort-oontext prefecence oprimisation
+ Ex, DD... p.r-Piowag Ni V..--i--laliiino
[e (Tahara 8) re (Rttogg B)l)
DPO:ORPO
sbort to to-long zoward alignment
Vanilla PO
mSo..oPO
(a) Comparison of objectives between original PO and SoLoPO. (b) Efficiency of Vanilla vs. SoLoPO. Wlong denotes the original Long-context input, and ishort denotes SoLoPO greatly cuts the original PO run the compressed chor--contett input preserving key task information. time and doubles the max trainable length.
Instract DLang-DMO SoLa-DPO
Task Performance
DLong-SimPP
$22
mS o-SimPO
ml ong--ORPO
60
11.
47.4
48-4
41 gt d86-010 ds
44/1
+4
40
31.
20
29.3
20
QA Lo-ggbeeecc
QAs RULER
LongDenchV2
NIAII Plus
Open LLM Leaderboard
Long-Contee Cong-Context Benchmarks
Short-Context Benchmarks
(c) Performance comparison of Owen2.5-7B 7-7-Intrrett 76 trained with original PO versus SoLoPO across various long-context and short-context benchmarks. Long-PO refers to original preference optimization on preference pairs sampled from long lexts, while SoLo-PO denotes preference optimization within our SoLoPO framework using preference data derived from compressed short texts.
Figure 1: Original PO vs. SoLoPO. (a) SoLoPO decouples long-context PO into two components: short-context PO and short-to-long reward alignment, reducing the complexity of preference data construction and minimizing long-text processing during training. (b) Under identical configurations, SoLoPO exhibits superior training efficiency compared to vanilla methods. (c) SoLoPO outperforms the original PO across various long-context benchmarks while maintaining short-context ability.
Tyef (loogg) with Tref Tshort)) to mitigate performance degradation on short-context tasks. However, LongPO is not generalizable to other PO algorithms. Accordingly, long-conter alignment poses three primary challenges: (1) difficulties in data construction, (2) inefficient training procedures, and (3) the absence of suitable optimization objective.
In this paper, we introduce Short-to-Long Preference Optimization (SoLoPO), general, simple yet effective framework to transfer the powerful understanding ability of short contexts (hereafter termed short-context capability) of LLMs to long-context scenarios. As illustrated in Figure Ial we first theoretically demonstrate that long-context PO short-context PO and short-to-long reward alignment (SoLo-RA). Intuitively, SoLoPO enhances the model's contextual knowledge utilization ability via short-context PO, while SoLo-RA explicitly encourages the model to align reward scores between outputs conditioned on long and short contexts containing identical task-relevant information, thereby improving its long-context ability. SoLoPO offers three key advantages over existing methods: (1) sampling preference pairs from compressed shortened long contexts improves data quality and construction efficiency; (2) applying SoLo-RA only to the chosen responses reduces the burden of long-text processing during training, leading to more efficient optimization; (3) the optimization objective accounts for connections between short. and long contexts, better favoring long-context alignment.
(33 be decoupled into components:
We apply SoLoPO to different PO algorithms, including DPO, SimPO 48). and ORPO 33) As shown in Figure Ic benefiting from SoLoPO, Sive2.5577-IInstt achieves better performance across various domains and lengths on QA tasks from LongBench [ and RULER [3I] demonstrating stronger generalization than models trained with vanilla methods.
trained solely MuSiQue (64
Results on LongBench [6] and the Open-LLM LLM-Leaderboard [19 further indicate that SoLoPO exhibits promising potential in handling contexts beyond pre-training window length, without
2

---

Published as a conference paper at ICLR 2026
promising short-context capabilities. Further analysis on NIAH-Plus 84 indicates that SoLoPO decoupled approach with explicit SoLo-RA notably improves the contextual knowledge localization capability of LLMs. Moreover, SoLoPO significantly enhances efficiency, enabling 2.1 x longer trainable length under ZeRO stage 555 with offloading while cutting run time by 52% and 26% for DPO and ORPO 9 length, respectively (Figure lb
Our main contributions can be summarized as follows:
We theoretically show that long-context PO can be decomposed into short-context PO and short-to-long reward alignment, providing new insights for long-context alignment.
We propose SoloPO,, general framework for long-context PO, which transfers the model's short-context ability to long-text scenarios while significantly improving training efficiency.
We integrate mainstream preference optimization algorithms into SoLoPO and empirically demon- strate LLMs can have much better performance within this framework.
SoLoPO: SHORT-TO-LONE PREFERENCE OPTIMIZATION
In this section, first introduce the background of preference optimization (PO), including DPO and the unified framework, generalized preference optimization (GPO) (62] ($ (...T). Then, by theoretically analyzing long-context preference modeling, we show that long-context PO can be decoupled into short-context PO and short-to-long reward alignment ($ 2.2). Based on this insight, we propose
SoLoPO and apply various preference optimization algorithms ($ 2.3
2.1 PRELIMINARIES
Reinforcement learning from human feedback (RHLF). RHLF [49] aligns LLMs with hu- man preferences through a two-stage process, further enhancing the model's capabilities. This involves training reward model rs that captures human preferences, followed by regularized policy optimization to align the L.M with the learned reward model, more formally as below
max we
[r. v)] 3Dxr. (=<[\=))[[[==>(((e))))
(I)
where my is the policy model, Tref is the reference policy, typically initialized with the SFT model.
Preference optimization (PO). Without explicit reward modeling, DPO [54] reparameterizes the reward function using the optimal policy model and directly models the preference distribution by incorporating the Bradley- Terry ranking loss (10), enabling single-stage preference alignment:
To(Uwl>)
To (z)
Lopo(TT; )res) = -E(x.ve...)> log
log
, log
(2)
Trey (yml
Teee (3) (z))
here, (yas y1) a preference pair. Furthermore, Tang et al. 62 propose GPO, a unified framework
for preference optimization, which allows us to parameterize the optimization objective using convex function f(-) and hyperparameters and ty:
C(ro, D) B(r.gunsu)> (f (n (rolz, Uw) - ro un) 7))]
(3)
2.2 THEORETICAL ANALYSIS OF LONG-CONTEXT PREFERENCE MODELING
Recall that a key challenge in long-text alignment lies in the inefficiency of data construction and training. Thus, can we represent long-context PO via short-context PO, thereby making data collection and training more tractable? We analyze the upper bound of general long-comtext PO loss, demonstrating the viability of this approach based on the redundancy hypothesis.
Redundancy hypothesis and compression rate. Redundancy, pervasive human language 69 59], while potentially aiding human comprehension, may adversely affect LLMs [0]. Particularly task-aware scenarios 332 42 74), for long context Clong and a task instruction 1, the model only needs to focus on relevant key content Cre! while ignoring irrelevant content cirr- Therefore, we can use compression rate 2 denoted as p, as a unified lens to observe long-context tasks, which refers to the information ratio between Crei and Ctong- Most long-context tasks, such question answering and information extraction, require only task-relevant excerpts from the source text B 32). yielding
p 100%. As special case, long-context translation 52 28 has compression p 100%.
3

---

Published as a conference paper at ICLR 2026
Problem setting. Regarding long-context scenarios, we use Plong Clongi I] to represent the input comprising the original long context Clong and task instruction I. Based on the redundancy hypothesis, Clong can be compressed, given the task instruction 1, into a context Gret that preserves all task-critical information. We denote by Ishort == Trel == [Cret; [] the concatenation of Grel and 7. For tasks with p < 100%, Tiohort is typically shorter than Flong; for p = 100%, they are identical. Given preference dataset ((stong.Bw..01)> the objective of long-context PO is to model the preference relation ply > yr Flong) by minimizing the preference modeling loss, as defined in Eq.
The upper bound of long-contes general preference modeling loss. To simplify notation in the following analysis, we define the preference loss in Eq. (3) for any given tuple (x1, 22, U1, U2) as:
In (21, 222 yı, 1/2) - [(n (x1. T (2 v2) -1)--
The expectation of Equation (4) can then be expressed as: ( Dzyi pn Dyz) = E2,, Pogii.:0 u2z In 1 (T1, T2: U1 92)]
(5)
Assumption (Discrimination of preference order). Based on the redundancy hypothesis, Tlong == [Cret, Carr); 1] contains more task-irrelevant information compared to Tshort == [Cret; 1. which may hinder LLMs' task performance [41 [50]. Consequently, distinguishing the order between yo and y given Ilong is more difficult than given Eshort (refer to Appendix 1.7 [.7 experimental evidence):
p(y= > y1 [ioong S p(y > y Ishort)
(6)
When Eshort and Flong are identical, the equality holds, giving a compression rate p of 100%.
Building upon the previous preparations, we establish the theoretical upper bound for the optimization objective over long-context data Theorem]] []] This bound provides formal guarantees that optimizing on short-context data while maintaining robust long-context performance is theoretically feasible.
Theorem 1 (Relation between contect and short-contex. preference optimization losses). Under assumption suppose I is a convex function and satisfies f(r + 7) f(-x + +7) S s((xl) for some function s( and non-negative constant 7 n.. Then the following inequality holds:
1 [.n.n 3 (Fahorz) DD..DD D, s(13n (Fo(#sshore v) T Falnng B))
E (Flong)
where ( Ly (Diseutt Drieni Dyw yu D Dyu yy wele)))))
The complete derivation of Theorem 1 is presented in Appendix 1.2 where s() is introduced with the primary objective of providing a metric to quantify the distance between re (Tahortt y)
and r (Xlong y). Given that s(-) serves as an upper bound, tighter instantiation is theoretically preferred; we provide empirical evidence for this claim in Appendix L.S] Additionally, an extension of Theorem is provided in Appendix I.5] which potentially holds promise for wider applicability.
Objective Function of Short-to-Lond Preference Optimization (SoL.POO). Based on the theo- rem we can define the general formula of the SoLoPO loss function:
3m [ro(x, vw) - r ( vn 31]
LSOLoPO
Ex SewDDeshrr seotr War DD yoot gi to suborr)
(8)
short-context preference optimization
+a Ez. DD.. ig..D (owrrr)) Vni mo (ys (eahor))
[s(3n) Ira (1 y) T (Xitnng v)1)
short-to-long reward alignment
Here, 17 and f()) are specified by the original PO algorithm. and s(-) satisfies f(r + +7)+ f(-r + S s(xx)) a hyperparameter balancing the two loss terms. Thus, we theoretically decouple long-context PO into short-context PO and short-to-long reward alignment. Specifically, we present detailed derivations of the SoLoPO objective from Theorem for two common convergence functions, f(x) x² and f(x)
T
log (æx,, in Appendices I.3 and I.4 respectively. Table 16 lists further examples of f() and their associated s(-). Moreover, the analysis in Section 4.2 provides
experimental evidence supporting the validity of this decoupling. When p 100%, Tiong and zshort are identical, rendering SoLoPO equivalent in form to the original PO, with differences confined solely to 1) and 7, for example, in long-context machine translation, the entire context is task-relevant.
4

---

Published as a conference paper at ICLR 2026
Short-to-long reward alignment (SoLo-RA). As shown in Eq. (9. SoLo-RA implies that, under optimal conditions, the reward model To should assign a consistent score to response y when
conditioned on either 2long or Zshorts as long as the input retains all task-relevant information Crel-
What does the SoLoPO learn? Unlike short-context tasks such as mathematics [58 39
that draw
upon the LLMs' intrinsic reasoning ability, long-context tasks necessitate a two-step process: first, identifying critical information within the given context, and second, executing the task based on that located information [38]. Consequently, proficient both contextual knowledge localization and contextual knowledge utilization reasoning. Compared to vanilla PO algorithms, which lack explicit modeling of the former, SoLoPO's decoupled optimization process is better aligned with these requirements, potentially leading superior performance due its distinct focus on: (a more
detailed discussion can be found in Appendix
Contextual knowledge localization. SoLo-RA (Eq. ( forces the reward model to implicitly predict Eshort ah [hoot( minimizing divergence between predicted Eshort and actual Eshort- In preference optimization, since the reward model is the policy model itself, this also improves the policy model's ability to identify task-relevant knowledge within long context.
Contextual knowledge reasoning. Since Zshort contains all task-relevant information, short- context PO (Eq. (8) enhances the model's ' reasoning ability over this contextual knowledge.
Non-decoupled short-to-long alignment. Based on the above discussion, another non-decoupled approach to short-to-long alignment involves directly applying preference pairs sampled from short texts for long-context PO or SFT, which we term Expand-Long Long-PO and Expand-Long Long-SF7, respec- tively. Experiments in Section 4.2 show that the decoupled approach yields superior performance.
2.3 APPLICATIONS OF SHORT-TO-LONG PREFERENCE OPTIMIZATION
Chosen-only short-to-long reward alignment (chosen-only SoLo-RA). Considering that yr ~ To(y) may not fully exploit task-relevant contextual information (e.g. responses "No an- swer"), performing SoLo-RA RA on y1 might introduce negative effects model learning. A supporting analysis of this issue is provided in Appendix [.1..2 Therefore, to further improve training efficiency, we only apply SoLo-RA Yw- Experimental analysis in Section demonstrates the effectiveness of this approach, which also reduces training resource consumption while maintaining training stability.
Table 1: Applications of SoLoPO to mainstream PO algorithms: DPO, SimPO, and ORPO. 1. Only the chosen-onie only SoLo-RA is shown; the short-context PO formulation identical to the original algorithms. 2. For DPO, Tref is omitted since it does not involved in differentiation.
Original Method
Reward
Chosen-only SoLo-RA
DPO
54
log (Mul)))) 8 feef (Mh (Me log ((x)
| log Te (yo (ashoot) - log To (yw [[ing))
SimPO 48
Iul log Na (1w
B log (T (short) 8 bul
log iTe ( (Ziong))
I (vr wsiong)
ORPO
(vwlle log 1 (xm
|log i i-= (aplllllaaaoo (ahhor))
salvu liahort)) log 1- (y (iinn)
SoLoPO can be applied various PO algorithms, provided that the corresponding convergence function f(-) and upper bound function s(.) are specified (see Table I6). We apply SoLoPO to mainstream algorithms, including DPO, SimPO, and ORPO, with their corresponding optimization objectives shown in Table I For brevity, we only present the expressions for the chosen-only SoLo- RA, while the objective functions for short-context PO remain consistent with the original methods. only align ( xz). See Appendix 1.12 for the complete derivation and expressions. Unless otherwise stated, SoLoPO refers to its chosen-only SoLo-RA variant in the remainder of this paper.
For DPO, since Trer (yw 2)) is constant and not involved in differentiation,
How does SoLoPO improve data sampling and training efficiency? As illustrated in Eq. S SoLoPO's preference sampling leverages Pshort, which, due to its shorter length and lower processing complexity compared to Tiong. enables faster and more effective sampling of high-qualit preference pairs. Furthermore, by applying chosen-only SoLo-RA, we process Ftong once and Sshort twice per training step. This contrasts with vanilla PO needing two Flong passes, where Flong processing is
5

---

Published as a conference paper at ICLR 2026
significantly more costly than Tshort- Thus, SoLoPO substantially boosts training efficiency. As the lengths of T'short and Flong become more similar (higher p). the efficiency gain from SoLoPO dimin- ishes. Further detailed analysis and potential optimization methods are discussed in Appendix H.S5
Table 2: Composition of different datasets and corresponding trained models. 1. SoLo denotes short-to-long alignment, where preference pairs derived from short contexts are used for long-contex: aligmment. 2. "*" means the corresponding PO method used in SoLoPO. 3. D is also utilized for training LongPO, which falls under non-decoupled Short-to-Loong DPO our framework.
Method
Dataset
Trained Models
D aot {(g, Tshoot Vie
dort) Diung {g. Dlongs )
SFT
short
M
D
dyet = {(g, Tshon, you 2o0t Vi
M
PO
D {(g, Floug Vi yone Vi )
M
Dett
expand long (q,, Viong Vilunn
M MSFI
Meggaed-logg long
SoLo
Des
expandsiongg = {41, Whong: shor
)
M Expurd-long
D Solo
{(g, Tobort, Blong shoet shom }}
(4)
SoLo
3 EXPERIMENTAL SETUP
Dataset Construction We construct shont and Flong from the MuSiQue 644 training set using the method in RULER 31. Specifically, we form long context by mixing relevant documents with random unrelated ones. On average, the short and long context contain 1.1K and 7.5K tokens, respectively. We use the original QA-pairs (g. a) from Musique as the questions and ground truth answers. To obtain preference pairs, we perform sampling with a temperature of 0,85 using the instruction model. For each input (Istton q, a). we sample N 32 Chain-of-Thoughle [67 outputs and then select the corresponding preference pairs (uatort _uf using the sub-em metric. Ultimately, we synthesize 5,000 training samples D = {(Ziong JTshcet q. a, yohort
-1 )) Additionally, we also
sample 5,000 real long-context preference pairs (y/ (glang 3/ long) based on Tlong. The composition of
different datasets is shown in Table 2 Figure 5 shows the pipeline and more details are in Appendix D
Baselines and Models As shown Table 2 we compare SoLoPO with other approaches that perform SFT or original PO on different datasets. Additionally, we incorporates results from LongPO [1I, which optimizes the reward of DPO based on short-to-long KI. constraint, replacing Trer Zlong) with Tref (y Issoon. We also introduce results from Qwenn.5--nsttruct322B// 1ammat.IInsstuuct
and for comparative analysis. We use Deven..--B-Innstuue (76) and Llama3. 8B-Instruct (34) as the backbones for our experiments with per-training context window of 32K and 128K, respectively (hereafter, 20we2..5-Instrus.. and Llama3. IInstruct are referred to as Qwen2.5 and 1 for brevity). Appendix H666 presents experiments on Qwen2.5- 14B to evaluate the scalability of SoLoPO. More training details are provided in the Appendix
Evaluation benchmarks To comprchensively analyze the effectiveness of SoLoPO, we conduct evaluations on both long-context and short-context benchmarks. The long-context benchmarks in- clude: (1) Real-world QA tasks from LongBenchVI (5), used to evaluate the generalization capability of different methods on multi- multi-document and single-document question answering in real scenarios within 32K context size. (2) Synthetic QA tasks based on RULER BII used to evaluate the generalization capability of different methods across various context lengths (4K/8K) ().K3.)) (3) We further leverage LongBenchV2 6 to analyze SoLoPO's potential on longer and more diverse real-world long-context tasks, and employ NIAH-Plus [84] to examine different models' context knowledge utilization ability in Section 4.2 For short-context benchmarks, we use MMLU-Pro (66].
MATH 27]. GPQA [57] IFEval 87. and BBH [60], following Open LLM Leaderboard [19].
we utilize Fl-score and multiple-choice accuracy as evaluation metrics, based on task-specific formats. For fair comparison, we select the best-pero checkpoint on LongBenchV1 within a single training epoch as the final model for cross- cross-benchmark evaluation. See Appendix E.2 for a detailed description of evaluation settings and benchmarks.
Following previous works 5 89

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTAL RESULTS
In this section, we present our main experimental results highlighting the effectiveness of the SoLoPO framework across various benchmarks and PO methods ($ 4.1I. Through comprehensive comparative analysis, we provide deeper insights into the key components of SoLoPO: decoupling and direct chosen-only short-to-long reward alignment and analyze the impact of the reward alignment
coefficient ($ 4.2) We then experimentally validate the efficiency advantage of SoLoPO ($
4.1
MAIN RESULTS
Table Performance comparison on QA tasks from LongRenchVI and RUELR. For LongPO, "pub" denotes the public checkpoint, while "reimp" indicates our implementation on D Bold and underlined indicate the best and the second-best performance, respectively.
S-Doc Vo M-Doc QA
QAs-RULER
Model
Avg.
4k
16k
32k
Avg.
Qwen2.5-725 ewen2.5-72BB. Instrucct
37.8
61.1
49.4
65.7
64.4
61.2
55.0
61.6
Bwve225-32B-Insttee
34.1
49.8
41.9
58.4
52.1
46.0
43.9
50.1
1ama331-70B-Instruu
28.5
64.1
46.3
72.1
68.8
52.4
23.2
54.1
LungPO-Qwen2.5-7BB5 LongPO-Qwen2.5-7B (pub)
34.8
52.6
43.7
62.4
54.4
48.9
43.1
52.2
27.5
38.3
32,9
54.7
51.9
40.6
45.9
Qwen2.5-7B-Innn -n-strtutt
Instruct
29.4
39.4
34.4
53.9
50.1
37.6
34.6
44.0
M
28.9
48.4
38.6
63.8
56.7
42.3
31.8
48.7
Mar
34.8
55.8
45.3
65.9
61.4
58.4
52.4
59.5
Mar
34.6
51.8
43.2
70.9
63.3"
45.3
46.9
56.6
M
35.7
58.2
46.9
71.0
64.2
60.6
53.2
62.2
M
38.0
57.6
47.8
66.4
64.5
62.7
57.7
62.8
Ma
34.7
53.6
44.1
70.8
62.5
42.T
48.8
56.00
Ma
34.2
54.9
44.6
69.8
49.1
47.2
57.5
Ma
58.6
48.4
69.2
66.0
62.7
63.9
28.9
48.4
38.6
69.1
62.1
50.8
46.6
557.1
Mag
34.8
55.8
45.3
64.9
59.9
50.0
45.6
55.1
M pare
37.6
61.4
49.5
70.8
68.3
64.0
57.3
65.1
- Instrurt
Instruct
30.3
49.3
39,8
58.3
49.2
42.9
35.6
46.5
MET
33.0
56.2
44.6
65.0
61.0
58.5
52.2
59.2
Mar
35.0)
57.3
46.1
63.7
59.0
57.5
53.5
58.4
M
33.1
55.T
44.1
64.0
59.3
59.7
30.4
58.4
Mase
35.4
55.4
45.4
63.2
58.9
53.2
58.9
Mas
35.2
57.3
64.4
60.1
SoLoPO effectively enhances long-context capabilities within pre-training windows. As il- lustrated in Table 3 SoLoPO achieves substantial performance gains and strong generalization, outperforming the original PO algorithm in 28 of 32 settings. Qwen2.5-7B, trained solely on the Musique (64) dataset, achieves a score comparable to Qwen2.5-72B on LongBenchVI Compared various original algorithms (DPO, SimPO, ORPO), attains performance improvements of 0.9, 4.3, and 10.9 points, respectively. Furthermore, SoLoPO exhibits consistently superior performance across varying context lengths on RULER, with only a performance drop observed at the length of 4K for DPO and SimPO. Similarly, we conduct experiments with SoLo-ORPO, the best-performing approach, on Llama3. --B, and the results further validate our claims. Given that Llama3. 1-8B has context size of 128K, 32K test length may already lie within its effective range (i.e. 25% (31), and our experimental data has a maximum length of 8K; therefore, the gains are smaller compared to Qwen2.5-7B. Specifically. SoLo-ORPO attains performance gains of 4.2 0.9 on LongBench and 10.0 vs. 2.4 on RULER over Long-ORE ORPO, for Qwen2.5-7B and Llama3. 1-8B, respectively. See Appendix H.6 for scalability experiments on Qwen2.5-14B.

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison of different models on LongBenchV2 and Open LLM Leaderboard. Red values indicate performance degradation short-context tasks compared the Instruct model.
LougBeedd/22
Open LLM
Overall
Ens
Hard
32L-128k
128k
MMLA-PPo
IFEval
HRH
MATH
GPQA
Arg.
Qwen2.5-770 70002------mmmrrutt
29.3(+0.7)
30.9
28.3
36,9
44.63
55.25
43.56
33.3(20.5)
32.2
T0
3366
15.
35.T5
"02T2""
10.0411444
92.0
788
35.7
25.9
26.7
46.74
71.46
55.85
35.55
51.45
47.90
1..1
36.
26.2
$5.0
25.3
27.8
44.78
74.70
55.26
38.21
50.95
48.78
M
30.1
24.5
33.8
22.3
23.3
46.6
73.61
55.37
52,12
33.2021
34.99
493.0
29.71 -007)
34.3
26.9
35.9
25.6
27.6
44.1
75,30
53.05
37.99
51.80
49.01
35.3 1 1.2
37.5
33.8
39.3
31.8
3500
44.66
73.86
54.78
35.57
31,03
48.12
27.1
23.1
29.5
21.2
4457
13.50
54.94
38.60
31.46
43.09
$4.89
Iamad. I-80-crd
Iestrnct
12.5(=1.00)
35.5
30.2
40.2
22.7
28.5
37.79
62.23
50:98
15.25
51.71
30.90
M
31.80 11..1111 1.77
34.5
30.0
38.5
28.5
27.0
36.37
60.31
50.23
17.90
51.79
39.32
30.91 +11
36.8
27.3
36.0
29.6
24.8
37.04
60.67
49.64
16.15
32.58
39.17
0.33
50.95
39.27
$4.5
30.04
99
SoLoPO shows better length generalization beyond the pre-training window. We test Qwen2.5 7B (w/ YARN (5I]) and Llama3.1-8P on LongBenchV2 with results presented in Table 4 For Qwen2.5-7B (w/ YARN), SoLoPO consistently outperforms the original PO algorithms across varying difficulty levels and context lengths. This highlights the superior generalization ability of models trained with SoLoPO, demonstrating its promise in handling longer and more diverse real-world long-context tasks. For Llama3. 1-8B, SoLo-ORPO shows improved performance across all evaluation dimensions. except for slight degradation on tasks with input length <32k words. While short-text training inherently aids length generalization [21,990] 90), SoLoPO's advantage likely stems from SoLo-RA, which explicitly enhances contextual knowledge localization,
discussed
in Appendix 1.9 Ablation experiments on NIAH-Plus in Appendix H.3 further support this claim. Additionally, DPO, SoLo-DPO outperforms LongPO(puh), despite the latter trained on larger volume of longer data. This may stem from LongPO's direct assignment of y T (long) y and y ~ I (y) shortt as yu without ensuring Vo > yt In contrast, our method constructs preference data from ground truth, ensuring correctness and quality.
SoLoPO maintains short-context performance. As shown To on the Open LLM Leader- board, SoLoPO maintains chort-context capabilities relative to the Instruct model, with only slight decrease on DPO. This trade-off is acceptable as SoLoPO simultaneously enhances the model's long-context understanding ability and training efficiency. See Appendix [.11 for the supporting analysis of short-context stability in Sol.oPO framework.
4.2
IN-DEPTH EXPLORATION OF SoLoPO
Long-
Both
Chosen Chosen-Only
Expand Long
SoLo-KL
SoLo-RA LLAMA
Qwen
LLAMA
Qwen
Improvement
20
10
12.5
13.4
13444.00
10.9
14.01..1
10
12.2
13.0 135
Improvement
8.6
14.12.55
15.1
20
10
12.8
10.4
0
20
15.5
17 77.88.8
18.8
13.5
19.29..9
11.1
19.2
21.1
20
13.7
140
20
12.9
16.4
10.8
9.1
20.41 SimPO
20.
21.21.111 ORPO
20
14.5
12.8
14.8
DPO
SimPO
ORPO
ORPO
DPO
ORPO
(a) Comparison of different SoLo-R RA approaches.
(b) SoLoPO vs. Approximation variants.
Figure 2: Performance improvements of different short-to-long preference optimization frameworks based on various PO algorithms over Qwen2.5-7B on LongBenchVI (top) and RULER (bottom).
8

---

Published as a conference paper at ICLR 2026
Empirical evidence for SoLoPO and superior performance of chosen-only SoLo-RA. We investigate the impact of chosen-only SoLo-RA versus applying SoLo-RA jointly to both yu and y (both), as defined in original SoLoPO (Eq. As shown in Figure 2a SoLoPO with both SoLo-RA consistently outperforms Long-PO, except for a slight drop on DPO, supporting the validity of its decoupling strategy theoretically established in Section 2.2 Moreover, the chosen-only SoLo-RA surpasses the both approach across diverse algorithms and models. Appendix H.2 reveals that, compared with the both setting, the chosen-only version yields larger reward margins (Figure [Oa) and assigns lower prediction probability to y (Figure TOb). These results suggest that chosen- only SoLo-RA achieves stronger fitting capacity and more stable convergence, leading better performance.
Direct reward alignment matters. To validate the effectiveness of SoLo-RA, we compare it with an approximation we termed short-to-long KL divergence (SoLo-KL):
(10) Here, we employ the absolute function to ensure non-negative training loss. From the reward expressions in Table I one can observe that SoLo-KL also promotes the convergence between To (Tatunt y and rg (Tongg ). For DPO and SimPO, SoLo-KL is equivalent to SoLo-RA, as the reward coefficients 3 can be integrated into the coefficient in Eq. ( As shown in Figure 2b for DPO and SimPO, the performance of SoLo-KL and SoLo-RA is comparable, with minor differences attributed to the slight variations in a. However, for ORPO, SoLo-RA significantly outperforms SoLo-KL on both benchmarks, demonstrating the effectiveness of direct reward alignment.
a Ez DD (mp kmm)) | log na (y (ahoon) log T [iang)]:
Decoupling long-context alignment yields better results. As shown in Figure 2b SoLoPO outperforms Expand-Long-POL a non-decoupled approach across different PO algorithms. We posit that the decoupled approach, through SoLo-RA, ex- plicitly improves the model's contextual knowledge utilization ability by enabling direct comparison between short and long contexts, while explicitly strengthening the model's perception of rewards and preferences. We further test the contextual knowledge utilization ability of different models (84) to validate the above claim. As shown in Table 5 SoLoPO achieves consistently greater improvements over Expand-Long-ORRPS confirming the rationality and effectiveness of our decoupled approach. Similar trends can be observed in the results of DPO and SimPO presented in Table 9 Additionally, Figure in Appendix presents heatmaps of model performance on NIAH-Plus when trained with different ORPO variants. Compared to Long-ORPO or Expand-Long ORPO, SoLo-ORPO significantly enhances the model's ability to retrieve information across various depths and context lengths in both single-hop and multi-hop settings, which further supports our claim.
Table 5: Performance gains of various ORPO
over Instruct model on NIAH-Plus 84
Model
S-Doc QA
M-Doc QA
AVG.
Qwen2.5-7B-Instruct
on NIAHI-Plus
M cxaasker
23.94
17.29
20.62
M TORN
25.98
18.83
22.41
-Lama..1-8B-Insrrue
M
MOMI
11.65
9.77
10.71
M piaoo
11.82
20.16
15.99
Impact of reward alignment coefficient a in SoLoPO To evaluate the influence of a in Eq. 9 we progressively adjust a in SoLo-ORPO across two distinct foundation models. As shown in Figure 3. all architectures exhibit characteristic response curves with clear peaks performance metrics, exceeding Long-ORPOR in most settings. See Appendix H.4 for analysis of DPO and SimPO.
4.3 EFFICIENCY ADVANTAGE OF SoLoPO
The chosen-only SoLo-RA RAA reduces the processing of long texts during training, thereby improving overall efficiency. As illustrated in Figure 4 we fix the length of Tshoo at 1 and investigate how varying lengths of Viony affect the performance gains and the efficiency gains in training time for SoLo-ORPO the Qwen2.5-7B setting using 4x A 100 GPUs Results show that as the length of Tlong increases, SoLo-ORPO achieves significant efficiency gains over the vanilla ORPO, cutting run time by 42% and 39% for Disng's 's lengths of 8K and 16K, respectively. Moreover, for Qwen2.5-7B with 32K pretrained context size, setting the length of Ishort and Ziong to 1K and 8K, respectively, yields a favorable trade-off between model performance and computational efficiency. Notably, as shown in Figure Tb] with only ZeRO stage 3 and offloading enabled, SoLoPO supports trainable
9

---

Published as a conference paper at ICLR 2026
Avreage Performance un
49.5 A
POOOeR
2710
Samp-Sol-o-RBR
235.98
20
48
SoL.oORFOOLLaa
Long-ORPO(
Long-ORFO((ween)
Run Raaii
240
210
180
O-
11.9
15.1
59%
15.0
179.20
15.5
15
Imporeenee 10 on
Len.Lannhhhhhh
145.
150
10
46
46.3
MM.21
4.2
&
54,00
66.63
0.1
0.6
3
5
8
10
IK
4K
12K
16K
Reward Alignment Coefficient
Length Long-Context (short-contex: fixed at 1k)
Figure 3: Performance w/ different a in Figure 4: Run time (RT) and performance gains (Imp.) SoLo-ORPO.
under varying lengths Tlong with Zishort fixed at 1K.
length up to 19K tokens, while vanilla methods are limited to 9K. See Appendix H.5 for more experimental details and discussions.
5 RELATED WORK
Long-Context Data Augmentation
This approach leverages advanced L.MMs to synthesize high- quality, denedeeend instruction-following data, used with PO SFT for long-context align- 91 75 4]. However, as text length increases, it becomes less reliable and efficient [63]
ment 443 80
Moreover, short-context alignment methods may underperform long-context settings [3 18 37
Long-Context Alignment Objective Optimization Fang et al. 18 propose LongPPL and LongCE loss to identify key tokens in long-text modeling and increase the loss weights for these critical tokens, thereby improving the effectiveness of long-context SFT. LongPO [1] searches for preference pairs based on short texts and applies them to long-context DPO training to achieve the non-decoupled short-to-long alignment discussed in our paper. Additionally, LongPO introduces short-to-long constraint, replacing Trer(y) (3/ Tlong) with Fye Tssunn, thereby maintaining the short-context ability. However, LongPÔ focuses context size expansion while its optimization is restricted to DPO. Both LongPO and LongCE suffer from limited training efficiency due to their reliance on long text processing, with LongCE incurring additional overhead associated with critical token detection.
SoLoPO introduces a theory-based decoupling strategy for long-context preference optimization, en- abling more effective modeling of contextual knowledge localization and reasoning. The chosen-only SoLo-RA variant improves performance while facilitating data construction and training efficiency, Moreover, integrating SoLoPO with long-context data augmentation may further improve its align- ment performance. A more comprehensive review of related work is provided in Appendix
6 CONCLUSION
this work, we propose SoLoPO, a general framework for long-context preference optimization (PO). Our method decouples long-context PO into short-context PO and short-to-long reward alignment, supported by both theoretical and empirical evidence. Experimental results demonstrate that the chosen-only variant of SoLoPO consistently outperforms vanilla PO methods and enhances the model's generalization ability in handling long contexts across diverse domains and lengths, while significantly improving both data and training efficiency. SoLoPO highlights the importance of the connection between short and long texts, paving the way for more effective long-context alignment.
Our findings open up several promising avenues for future investigation, such as enhancing training efficiency for fully context-relene tasks and exploring how the core ideas of SoLoPO can inform the optimization long-output generation tasks. more detailed discussion provided in Appendix C
ACKNOWLEDGMENTS
Yang Gao supported by the Major Research Plan of the National Natural Science Foundation of China (Grant No. 92370110) and the Joint Funds of the National Natural Science Foundation of China (Grant No. U21B2009).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on optimizing preference learning objectives for large language models (LLMs) in long-context scenarios. It does not introduce unexpected ethical risks beyond those commonly considered standard NLP research. Although LLMs are trained on large amounts of Internet text that may contain harmful content, our study targets long-context understanding rather than direct deployment, which greatly reduces the risk of propagating biased information. All models and datasets used in experiments are open-sourced and publicly available, ensuring transparency and minimizing potential ethical concerns.
REPRODUCIBILITY STATEMENT
We have provided comprehensive details of our method (Section 2.2 and Appendix Dr. data synthe- sis pipeline (Section D) model training settings (Appendix E.I and evaluation benchmarks and configurations (Appendix E.2) in the main paper. All datasets and models used in this work are openly available, with direct access links provided in our paper and the supplementary materials. To further facilitate reproduction of our results, we release the complete source code, data examples, and step-by-step usage instructions in the supplementary materials. These resources are intended to enable other researchers to fully replicate and verify our experiments.
REFERENCES
[1] Mohammad Gheshlaghi Azar. Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR. 2024.
[2] Yu Bai, Heyan Huang. Kai Fan. Yang Gao. Yiming Zhu, Jiaao Zhan, Zewen Chi, and Boxing Chen. Unifying tross-lingual summarization and machine translation with compression rate. In SIGIR 2022 Proceedings the 4Sth International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022 Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1087-1097. Association for Computing Machinery, Inc, July 2022. doi: 10. 1,5514447779955355 7553777077 Publisher Copyright: @ 2022 ACM.; 45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022; Conference date: 11-07-2022 Through 15-07-2022.
[3] Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, and Jackie CK Cheung. CItruS: Chunked instruction-awaree state eviction for long sequence modeling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Em- pirical Methods in Natural Language Processing, pp. 5908-5930, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 11865//1/2224.mm/ll 338..338. URL https: http://acllnnhhoog...oooo org 2024. mmnlp-maimn 338/
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Qi, Lei Hou, Jie Tang. Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1376- 1395, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 18.13v53/v1 2024. finddiind emnlp.74 74. /Rtpstts:
2ttg/2024, http://22...dnnass-eeee..e... windings-ennn.a... 747
[5] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang. Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: bilingual, multitask benchmark for long context understanding. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the A ssociaaion for Computational Linguistics ( (Volume I: Long Popers), pp. 19-3137, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 12053//1/2024aacllllo
172. URI. anw.aaaannoooooooooooooooooooooo ogy. or cr 2024 aaw.0.4441111... ac1 1ong. 72./
[6] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng. Xiaozhi Wang, Xin Lv. Shulin Cao. Jiazheng Xu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longhench v2: Towards deeper understanding and reasoning realistic long-context multitasks, 2025.
15204
[7] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng. Siqi Zhu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context LLMs. In The Thirteenth International Conference Learning Representations, 2025. URI. httppsh//ppenneeeeeee ew net forum? i
kQ5s kYhwwl
11

---

Published as a conference paper at ICLR 2026
[8] Masha Belyi. Robert Friel, Shuai Shao, and Atindriyo Sanyal. Luna: A lightweight evaluation model to catch language model hallucinations with high accuracy and low cost. In Owen Rambow, Leo Wanner, Marianna Apidianaki. Hend AI- Khalifa, Barbara Di Eugenio. Steven Schockaert, Kareem Darwish, and Apoory Agarwal (cds.), Proceedings of the st International Conference on Computational Linguisties: Industry Track, pp. 398-409, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL.ttps:: acl www.alonnnoooooooooooooooooooooe org/ www.nggiiii...o -ii.ggginnnurry.. .347
[9] Baolong Bi. Shaohan Huang. Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun. Qi Zhang, and Shenghua Liu. Context- dpo: Aligning language models for fantext-fattfulnesss. CoRR, abs/2412.15280, 2024. URL BRLhttps
18888/5000.i... 2412. 15280
[10] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-44 1952. ISSN 00063444, 14643510. URL http
/www. jstor. www.sttrrrrrgseoeeeeeeee wtt.er/g//ttaa/eeeeeeee./em
[11] Guanzheng Chen, Xin Li, Michael Shieh, and Lidong Bing. LongPO: Long context self-evolution of large language models through short-to-long preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL /htt:p::/openreviiee
not forum? ? L 31.m
[12] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian I Liu, Song Han, and Jiaya Jia. LongloRA: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https::s:: /ttp:////oeeeeiveeeeeeee
nntt/ forum? i 6P6PMJ fdak
[13] Tri Dao. Flashattentionn-:: Faster attention with better parallelism and work partitioning, 2023. URL https: https::/arxxi..rr///bbbss
08691
[14] Pradeep Dasigi, Kyle Lo. Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seekinn qucstions and answers anchored in research papers. In Kristina Toutamova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkami-Tuur Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (cds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610. Online, June 2021. Association for Computational Linguistics. doi: 1o.18853/v1/202111 waw.022llaaaamammmaaaa.a URL
2021 hattp::/aclanthologg 2002 wawp:/22011.mccmamaaa www.naancllaaam maa.aaclmmmaii..3.3 365/
[15] Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey on long text modeling with transform- ers. ArXiv, 14502,, 2023. URL https: http://///aps.eeetiicccciccci www.oopuuroooooooooll
257232619
[16] Zican Dong, Junyi Li, Jinhao Jiang, Mingyu Xu, Wayne Xin Zhao, Bingning Wang, and Weipeng Chen. Longred: Mitigating short-text degradation of long-context large language models via restoration distillation, 2025. htt.:tttt.///aaa.oorr////// abwpg//////////a/ http://aabb/222.2223 07365
[17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. Ahmad Al-Dahle. The Ilama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL htttps://aaatt http://ppp.semaa....em.eemm semant icscholar
org/ org/Corpuse Corpus ID: : 0011221151115
[18] Lizhe Fang. Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? In The Thirteenth International Conference on Learning Representations, 2025. URI.
net./
forum?id f 1lA.kSStM.M..
[19] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open 1lm leaderboard v2. httt::ttt////eeett httttt:e//uggggggaccc... wttp://ccppppppppppeeeeeeeeeseee www.:ennnnn---aacbbaaaeeaaaaeaea oper_
1mm Icaderboard 2024.
[20] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
https://openrevicee net/ www.:oooommmdmaame
[21] Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How train long-context language models (effectively). Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63d Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7376 7376-7399 Vienna, Austria. July 2025. Association for Computational Linguistics. ISBN URI. hetps: actps://ast //acl anthol org
891--897771--99 doi: 10.1
2025. acl - 1ong. 3667
12

---

Published as a conference paper at ICLR 2026
[22] Tianyu Gao, Alexander Wettig, Howard Yen, and Dangi Chen. How train long-context language models (effectively), 2025. httl:t/pp.///opeeeeeeeeeeeeo////
modelinn [23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Confer- ence on Language Modeling, 2024. /t//httpss//ppperreiiiieeeeee/ep
VY2
[24] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL wRt.ttts:tt///tt httptttt///ppeeeieer
net/forum? www.yLooouyyl___...
[25] Alexander Gurung and Mirella Lapata. Leaminng to reason for wow--oooooooooo story generation. In Second Confer- ence on Language Modeling, 2025. URL http://sa///geeeeee....eeee/eeee
Pid=dr 30g5chR2
[26] Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang. Jingjing Xu, Zhi Zhang. Hongxia Yang. and Di He. Two stones hit one bird: bilevel positional encoding for hetter length extrapolation. In Proceedings of the s1 International Conference on Machine Learning, ICMI.224 JM... 2024.
[27] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv www.://rg//b/bbaab 03887774
[28] Christian Herold and Hermann Ney. Improving long context document-level machine translation. In Michael Struhe, Chloe Braud, Christian Hardmeier, Junyi Jessy Li, Sharid Loaiciga, and Amir Zeldes (eds.), Proceedings of the 4th Workshop on Computational Approaches to Discourse (CODI 2023), pp. 112-125. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18633/v 1/2023.codi- 1.15. URL httt::s///aclaathhooogooooo org/ codi-iii-1 1.15/
[29] Xanh Ho, KhoaKhoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengging Zong (eds.), Proceedings of the 28th International Conference on Computational Lingwistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Commiltee on Computational Linguistics. doi: 10. 11865//1/2020.coliiii moini-oaiiinn---550000000 580. URL BBRl:/ttte//eeclaaaboooooorrrr// wtt../oooo/......ooooo...o cwwiinggoniooooc...c..o main
5807]
[30] Jiwoo Hong, Noah Lee, and James Thorne ORPO: Monolithic preference optimization without reference model. In Yaser Al-Onaizan Mohit Bansal, and Yun Yun-Nung Chen r. Proceedings of the 2024 Confer- Empirical Methods in Natural Longuage Processing, pp. 11170 11189,,19,,111111111111111111111 Miami, Florida, USA,
November 2024. Association for Computational Linguistics. doi: 11886///1/222242mmmpmmaaaa
URIttttps::tt / anntho wwwnthooooy...oooo wow.oooy...o....o....oo.o....... /2024, .en:4.mmmnn1p emaan n. 6267
[31] Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's s the real context size your long-context language models? First Conference on Language Modeling, 2024. URL httth:tp////// http::///ppppeieeeeeeee net Eorum? httpe//ooommmdmioioeoeee -id=kIoBbibbb 765y
[32] Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Fan Yang. and Mao Yang. Fewer is more: Boosting math reasoning with reinforced context pruning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13674-13600 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 12644//1/02244.mmmmmmmmmaaa 758/
URL http:p/:///claaaoooggyyy .rg. 20044 emnle-mlm.e..... maiin
[33] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations enabling training extreme long sequence transformer models, 2023.
wttp:////////../......o/ 2309.145099
[34] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed, Mistral 7b, 2023. URL https:t/taaaatttt org/ 06822
[35] Tomá's Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10. 162/tacl___000222 URI. htt.httpa
//acl hol www.ooogg................ 818 1033/
13

---

Published as a conference paper at ICLR 2026
[36] Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmiiry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Iims with long context reasoning- -annng------ a-aaastaccc. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 106519- 106554. Curran Associates, Inc., 2024. URL h neur. wwwpppppppppppppeeeeeeeeeeeeeee Feseeeeeei/ httpp/rrrrrssspppeeee/eee httpp/aapprr////////// zile/ pdf
and Benchrarks. TTrakk.
[37] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to -oe-uuee-eeeeeeeeeeeee pre-training for natural lan- guage generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics, 2019. URLpstps: api. semant icscholar wwt.:orggo/oooooeeeeo IC: 204960719
[38] Jiawei Li. Yang Gao. Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li. Huashan Sun, Yuhang Liu. Xingpeng Si, Yuhao Ye, Yixiao Wu, Yiguan Lin, Bin Xu, Bowen Ren, Chong Feng, and Heyan Huang. Fundamental capabilities and applications of large language models: A survey, ACM Comput. Surv. 58 (2), September 2025, ISSN 1360-0300. doi: 10. 10.174573736632. htttss::://00 org.1/....r.... 1145/
3735632
[39] Jiawei Li, Xinyue Liang. Junlong Zhang, Yizhe Yang, Chong Feng. and Yang Gao. Pspo*: An effective process-supervised policy optimization for reasoning alignment, 2025. /ttttsss//aaxxii org/
abs/2411 11681
[40] Yanyang Li, Shuo Liang, Michael Lyu, and Liwei Wang. Making long-context language models hetter multi-hop reasoners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2472-2475. Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 20.18853/v1/2024, acl-long 135, URL httpstts: / ac www.acaanooooooooooooooo org. 2024 2
1357
[41] Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6342-6355 Singapore. December 2023, Association for Computational Linguistics. doi: 10.18653/v1/2023 www.:222eemmlp-mmmmemme 1 /ttttttss///aaaattttttte wwwphhoooooooooo/oooeooo /2033, eennpp-main. 391/
[42] Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: V LLM knows what you looking for before generation. In The Thihty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL. URI.https https? http//ooummmdeeee
//openreview.
4G0q21
[43] Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, and Arman Cohan. Mdeure: scalable pipeline for multi-document instruciion-following CoRR, 13.2444,, 2024. doi: 10.48550/ARXIV. 2410.23463.
(tRtsstts::/ /c org/10. 48550/arXiv 2410. 23463
[44] Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Moore Wang. Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijic Zhu, Junlan Feng. Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Ying Tan, Zili Wang, Jian Yang. Wei Ye. Bo Zheng, Wangchunshu Zhou, Wenhao Huang. Sujian Li, and Zhaoxiang Zhang. A comprehensive survey on long context language modeling. 2025. URL httph:ht////. : api Corpus 27:27 1533
org/
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL/https: httpctps:///oerzcvviiii http://pwwrrooooo.oooooorroro..t
[46] Enzhe Lu. Zhejun Jiang, Jingyuan Liu, Yulun Du Tao Jiang. Chao Hong. Shaowei Liu, Weiran He. Enming Yuan, Yuzhi Wang. Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu. Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan. Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang. Xinyu Zhou, Mingxing Zhang, and Jiczhong Qiu. Moba: Mixture of block attention for long-context llms, 2025. URL https: https://arxiv.. org/ abs 2502 13189
[47] Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale, 2023. httlhttps///arii org/ abs tos/2309..m 12012399946656
[48] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-fiee freee reward. In The eiihty- Annual Conference Neural Information Processing Systems, 2024. URI
https://open net wttt:/oormmiddd1100 11kb
14

---

Published as a conference paper at ICLR 2026
[49] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrewe
net/ forum --i.ddacGGKK
[50] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang. Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang. Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful aans-sgoo prompt compression. Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 963- 981, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2024. 0024.fndiing--al..33
httts::ts: / htt//aaaaatttt wow.oooooyyooooo wrg. Findings-acl... .57
[51] Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window
extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL URL Bttp:ttts///:peeeettte
net. wtt.rrummi.iaaBBBeeeeea
[52] Ziqian Peng, Rachel Bawden, and François Yvon. Handling Very Long Contexts in Neural Machine Translation: a Survey. Technical Report Livrable D3-2.1, Projet ANR MaTOS, June 2024. URL https
//inria.. hal. www.scceeeeeaaaaeecc 6555528
[53] Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, xu Zhao Pan, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobateh: Lossless training speed up by unbiased dynamic data pruming. In The Twelfth International Conference on Learning Representations, 2024. URL
https://apenrevier. net Iorum htt.:/ooomnnmmddd..m.e
[54] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18890 2023. UtL
semant ieschholrrrrrcoccc wtt.sschhl.a...ooooooooooss/seae 25895932 959321
[55] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL perttps://axxiv.oot www.:xiv..oooraaoooooo abs/ 11900 02054
[56] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su. Kevin Duh, and Xavier Carreras (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 18613/86 1/1/3 1264. URL https://aclanthir Logy. orgp:/6-1000000600000
[57] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
httl:httss///aaxt http:///awaxxxoo///////////
1. 12022
[58] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. (tttatts::// httppp::///xxx...rra/ abs 2402 03300
[59] Neil J. A. Sloane and Aaron D. Wyner. Prediction and entropy of printed english. 1951, URL https //api. semanticscholl wwtp:/rr...oro/ooouoooooo somatttiGeeeodlrrr.oo///rooppppp
9101213
[60] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big bench tasks and whether chain-of- thoughh can solve them, 2022. JRLhttpss:/ / //aaxiivvvv.. org/ abs /asb///011
09261
[61] Xinyu Tang. Xiaolei Wang. Zhihao Lv, Yingqian Min, Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. Unlocking general long chain-of-thought reasoning capahilities of large language models via representation engineering, In Wanxiang Che, Joyee Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 633rd Annual Meeting of the Association for Computational I. Linguistics ( (Volume I: Long Pupers). pp. 6832-6849. Vienna. Austria. July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0 doi: 10.1165531122225.ddllll aoc./ooo.....ll-o.333339.33 URL https:
/ http:///aaaaatttlgy...ooo
2025.ac/ long. 3397
[62] Yunhao Tang. Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference on Machine Learning, 2024. WRRlhhttps://
ew. net. forum? ddggu3accAAa
15

---

Published as a conference paper at ICLR 2026
[63] Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. LOGO long context aligument offficientpefferennee via efficient preference optimization. CoRR, abs/2410. 18533, 2024, doi: (0.4850/ARXX..240.0155
/tt/htts:///do.rrg/// 488500/a2511y 2410. 18533
[64] Harsh Trivedi, Niranjan Balasuhramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguisticy, 10:539-554, 2022. doi: 10.1162/tacl a 00475. URL https: ht..:ps:/aallantooooyy.. org/
2022. tac1_..371
[65]
Minzheng Wang. Longze Chen, Fu Cheng. Shengyi Liao, Xinghua Zhang. Bingli Wu, Haiyang Yu. Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang. Fei Huang, and Yongbin Li. Leave no document behind: Benehmarking long-context LLMs with extended multi-doe QA. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 5627-5646, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/v 1/253// main.322. URL htttpt/:://// http://aaaaanntooo...ooooo
emnip- ma ma........................ 3227
[66]
Yubo Wang. Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren. Aaran Arulraj, Xuan He. Ziyan Jiang, Tianle Li. Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. wttpbccps://arxxv.orrgg www.oor///aabaaaaeoooooo..eeaa/s a40eaa///222020 01574
[67] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems,
[68] Wikipedia contributors. BallbackkLLibblee divergence. https:://nn.wkkipp crg/wiki/
ottpaa/ceeosssteeeeeeeeeeeeeeeee 3024-06-15 10.
2024.
Accessed:
[69] Emst C. Wit and Maric Gillette. What is linguistic redundancy. 2013.
URL https://api
semanticschoun
orgg/ooxpusI::142222 14256655
[70] Tong Wu, Yanpeng Zhao, and Zilong Zheng. An efficient recipe for long context extension via middle- focused positional encoding. In The eiirth Annual Conference on Neural Information Processing
Systems, 2024, BtLpstt:::/ http://ppwwweeeeeeeeeeee net. /tteee//oooeeeeee forum? a MSSON
[71] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
URL
eww net forum? 2 iiee2idd-n3 d-NG7 2VF
[72] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 4663, Mexico City, Mexico, June 2024. Association for Computational Linguisties. doi: 10. 18653/v1/2024, naacl- long. URL https:tts:///ett ant
nww.caaala- wow.onnn0000 .260/
[73] Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 1866):186377, 2024.
[74] Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmentte LMs with con- text compression and selective augmentation. In The Tweifth International Conference on Learning Representations, 2024. /tt.htttt:://peereeea wwwpiiiwwweeeeeoooeeeeee forrm http:/orrmmmmmmmm11iiiiiiii.au
[75] Cehan Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Anfan Liu, Hui Xiong, and Jian Guo. Longfaith: Enhancing long-context reasoning in imms with faithful symthetic data. CoRR, 2ttps/2552.122233 2025. doi: 10.48550/ARXX1.25022.2555 12583
URL https:/ http:oo..ooooooo/ org. 18..48.5//a.iii...
[76] Qwen An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong. Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren 7hou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia,
16

---

Published as a conference paper at ICLR 2026
Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang. Yunyang Wan, Yugi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report ArXiv, abs/2412.15115,, 2024.
[77] Yizhe Yang, Huashan Sun, Jiawei Li, Runheng Liu, Yinghao Li, Yuhang Liu, Yang Gao, and Heyan Huang. Mindllm: Lightweight large language model pre-training, evaluation and domain application. Al Open, 5: 1-26, 2024, URL puttps://apl semant icscholar otg/Corpu ID: 2011221111111
[78] Zhilin Yang. Peng Qi, Saizheng 7hang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and annichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369-2388 Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v110 1/D99-1259. URL htttp::/aclanthho
wwg. 12597
[79] Jiajie Zhang. Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longcite: Enabling LLMs to generate fine-grained citations in long-context QA, 2024. URL http:p:::///peeeeviieeee
net forum? BB
[80] Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao. Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong. Ling Feng. and Juanzi Li. Longreward: Improving conteco large language models with Al feedback. CoRR. 2bs/2410.21252, 2024. doi: (0.48550/ARXI/2240022222 URL doi org/ 10. 48550/
arxiv 24 10. 21252
[81] Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. Extending llama-3's 's context ten-fold overnight. CoRR, 0aw222444195555 2024. doi: 10.48550/ARXIV. 2404. 19553. URL U httpt::t///cct http///...rrrg///////////// 48550/arxiv 2404. 19553
[82] Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context com- pression with activation heacon. In The Thirteenth International Conference on Learning Representations, 2025. Utt.htttss///peereeeeeee
ew. net. www.//orummiiccc 6= eQT FO2
[83] Xinghua Zhang. Haiyang Yu, Cheng Fu, Fei Huang, and Yongbin Li. Iopo: Empowering llms with complex instruction following via input-output preference optimization, 2024. URL BtLLhttps:// / org/
abs/2411, 06208
[84] Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. LONGA- GENT: Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 16310 16324, Miami, Florida, USA, Novem- ber 2024, Association for Computational Linguistics. doi: 118666//122224ccm// m1in.912. URL
https: / http//clannthoooooooooooooo org/ 2024. complp-maing 912/
[85] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human foodback. arXiv preprint Xrvv:2305 10425, 2023.
[86] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified effi- cient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demon- strations), pp. 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 118853v1/2024..... acw.:24444.lldeoo.....
bttps:: wnt.oloo......oooooo.oooo org/ 2024 acl - demos. 38/
[87] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cestruction-followiin evaluation for large language models, 2023. URL https: :/arxii /arxiv
org/ abs/23133 07911
[88] Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of LLMS via positional skip-wise training. In The Twelfih International Conference on Learning Representutions, 2024. URL https:://penzeeee/
bet/foorne id=
32 qx
[89] Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. Ch matters: Improving long- contect language models with rcasoning path supervision. CoRR, 20090 2025. doi: 10.48500ARRXIV25502..099 URL
https://dci.orgg 10. 48550/ 2502. 2079C
[90] Wenhao Zhu, Pinzhen Chen. Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Ffffective data synthesis for long-context instruction tuning. CoRR, abs/2502. 15592,, February 2025. URL hollhhhtps:///o/ .org 18550/0000i110 2502. 15592
17

---

Published as a conference paper at ICLR 2026
[91] Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan. Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Effective data synthesis for long-context instruction tuning, 2025. URL
15592
[92] Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan. Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Effective data synthesis for long- long-context instruction tuning. ArXiv, abs/2502. 15592, 2025. URL _
semant cscho ar wtt/Corpus D:2765:7 02227557766
A THE USE OF LARGE LANGUAGE MODELS
We explicitly disclose that large language models (LL.Ms) were used solely for the following purposes: (1) Writing refinement limited to minor grammar correction, wording improvement, and stylistic polishing the manuscript text; (2) Data generation specifically, the preference data required for our experiments were sampled from the corresponding open-source Instruct models, following the procedures described in the paper.
LLMs were not used for idea conception, methodological design, result analysis, any other substantive scientific contribution to this work. All research ideas, methodological innovations, experimental executions, analyses, and conclusions were conceived, implemented, and validated entirely by the authors.
B RELATED WORK
Numerous studies focus on extending the limited pre-training context windows of LLMs to support longer inputs, including post-training long-context corpora 20
12 81), designing novel architec- tures [24] 46 23). or modifying positional encodings [2]]]] 70 88 However, researches reveal that the capability within the pretraining window of current LLMs has
been fully activated, resulting in suboptimal performance on long-context tasks [31 36 8 79.. To address this challenge, existing approaches primarily focus on two aspects: data augmentation and training objective optimization.
synthe- sizes high-quality, long-dependency instruction-follovingg data for supervised fine-tuning or offline preference optimization. Instruction synthesis [3 4 directly leverages one multiple real long documents to prompt advanced LLMs to generate diverse instructions and responses for long-tes alignment. Context synthesis 991, on the contrary, is built on real QA data by having models synthesize background contexts based on questions, then randomly concatenates multiple synthetic contexts to create long-context instruction-folloving data. Although data augmentation demonstrates some effectiveness, directly applying short-context training methods to the long-context setting may overlook differences between short and long contexts, thus failing to fully activate the model's potential capabilities [5 37). While SoLoPO incorporates the connection between short and long contexts into its training objective, it can be combined with data enhancement techniques, which has the potential to further enhance model performance.
Long-Conters Alignment based on Data Augmentation Most research 43
80
75
Long-Context Alignment based on Training Objective Optimization Some works explore optimizing training objectives to further enhance long-context capabilities in LLMs. Fang et al. [18] propose LongCE loss to identify key tokens in long-text modeling and increase the loss weights for these critical tokens, thereby improving the effectiveness of long-context SFT. LOGO (63 employs multiple negative samples and adapts the SimPO objective to minimize the probability of generating various dis-pree dis-prefeeeece instances. LongPO [T] searches for preference pairs based on short texts and applies them to long-contex: DPO training to achieve the non-decoupled short-to- long alignment discussed in our paper. Additionally, LongPO introduces a short-to-lone constraint, utilizing the output distribution of short texts on the reference model as a reference during long-context DPO training (replacing Tree (y Iitong) with (ye #shors)), thereby maintaining short-context capabilities. Although LOGO and LongPO adopt similar data construction strategies ours, they fundamentally differ in that they do not decouple the optimization objectives. As a result, these methods fall into the category of non-decud short-to-lone lont- alignment discussed in our work. Moreover, LOGO, LongPO and LongCE suffer from limited training efficiency due to their reliance on long text processing, with LongCE incurring extra overhead from critical token detection.
18

---

Published as a conference paper at ICLR 2026
SoLoPO introduces theoretically grounded framework for long-context preference optimization. Specifically, SoLoPO explicitly models the connection between short and long contexts, decoupling long-text preference optimization into short-text preference optimization and short-to-long reward alignment. The chosen-only variant of SoloPO not only improves the model's long-context ability, but also significantly enhances the efficiency of both data construction and training procedure.
C LIMITATIONS & FUTURE WORK
Training Efficiency Enhancements For tasks where compression rate is 100%, such long- context machine translation, SoLoPO is equivalent to the original PO algorithm, offering no gain in training efficiency. Given that redundancy also exists in the hidden states of LLMs B2 50 4]] future research could extend token-level compression to hidden-state-level compression, potentially by combining our approach with KV-cache compression techniques 22 Such extension would better support a wider variety of long-text applications. Moreover, although SoLoPO leverages the chosen-only SoLoRA strategy, it remains necessary process long sequences, which can lead to efficiency bottlenecks when dealing with large-scale datasets. Future work could explore the
decoupling strategy of SoLoPO combination with data pruning techniques [53 47. aiming to
appropriately reduce the processing of long-context inputs and thereby improve training efficiency.
Toward Extended Theoretical Analysis SoLoPO is primarily designed for long-context input scenarios, and therefore does not directly address the challenges of long-text generation [7]. Extending our theoretical analysis to long-text generation settings represents a natural and important direction for future work, which would further broaden the applicability of SoLoPO. The core principle of SoLoPO posits conditional equivalence between short and long inputs. We believe that this concept similarly extends long-output generation tasks, as exemplified below:
long CoT that ultimately yields correct final answer is equivalent to a concise CoT achieving the same result from a task completion perspective.
Long Chain-of-Thoon Th-ouuhhtt (CoT) 61
Text Refinement: a plainer, shorter version, as long as the core semantic meaning retained.
stylistically refined and thus longer text can be deemed semantically equivalent
Story Generation
25 Longer story outputs are functionally equivalent to shorter versions if both fulfill a user-spe narrative arc or core plot, despite offering greater descriptive depth.
Moreover, observed from Equation (9), original preference optimization (short-context PO) focuses on discrepancies in the output space, whereas our proposed SoLo-RA emphasizes relationships in the input space. This raises intriguing question: Can the decoupled preference modeling approach underlying SoLoPO be generalized to other tasks where modeling input-side connections plays critical role? (other context-aware tasks, such as complex instruction following (83) and context- faithful alignment [9]) Investigating this direction may yield new insights into the design of more expressive and flexible preference optimization frameworks.
More experimental analysis Due to resource limitations, our current experiments primarily focus on efficiently activating capabilities within the model's pretrianing context window (32K). ). Fu- ture work should further evaluate the effectivenness of SoLoPO on even longer context and larger model scales to fully understand its capabilities. Additionally, SoL.PO introduces two hyperpa- rameters conterssioomm ratio and reward alignment coefficient a--hhih require manual tuning. Future work could explore automated methods for determining optimal values for these parameters. Moreover, while we believe that SoLoPO could support self-evolving mechanisms for progressive context window expansion I1 633. this remains to be validated via more comprehensive analysis.
Higher-Quality Data Synthesis While our current approach constructing the short-to-long dataset is simple yet effective, it suffers from limited realism and diversity. Future work could explore integrating SoLoPO with existing data augmentation techniques to synthesize more realistic long-context instruction- followiinn data, such as instruction or context synthesis grounded in authentic Moreover, extending the dataset to cover a broader range of long- text scenarios such long-documee summarization, long-in-context learning, and long-form dialogue understanding- could provide a more comprehensive improvement of models' long-context processing capability.
data sources 43
91 75
19

---

Published as a conference paper at ICLR 2026
D DATASET CONSTRUCTION
In this section, we present the methodology for the short-to-long preference dataset construction. As noted in our preliminary experiments and related studies [9] (2], training on shorter texts can still yield improvements performance over longer contexts. Inspired by these findings, we heuristically set the average length of short contexts (Eshert) to approximately 1 K tokens and long contexts (Zlong) to around 8K tokens, which corresponds to 25% of the pretraining context window (32K) of Qwen2.5-7B In--sructt.. The overall data construction pipeline is illustrated in Figure 5
prompt (short)
2 inferenice sampling
rull- rull--aased basedd selection
short context
question
response candi
correct answer
random sample
X
irrelevnnt docs
add distractor
LLM
preference
SoloPO
long context.
short-to-long detaset
Figure 5: Illustration of the construction pipeline for the short-to-long dataset. (1) Irrelevant docu- ments are randomly sampled and concatenated with the original short input to form long contexts. (2) Multiple candidate responses are generated based on the short context and question via the instruct model. (3) Preference pairs are curated using sub-emppased selection guided by ground-truth answers. (4) The final short-to-long dataset, composed of short contexts, long contexts, questions, and preference pairs, is used for training LLM with SoLoPO.
Context Synthesis We follow the strategy proposed by RULER 31 to generate synthetic contexts based on the MuSiQue dataset. Specifically, given an original sample (g, a, Dsupporting) where g and denote the question and ground-truth answer respectively, and D Deupporting is the set of supporting evidence documents, we synthesize a context of target length L as follows: we randomly sample a set of irrelevant documents Dirrelevant- such that the total token count of suuportinng U Dirrelevant approximates L. The token count is calculated using the tokenizer of the corresponding instruct model. In this setup, both short and long contexts contain the necessary information to answer the question; however, the long contexts include more distractor content, thereby simulating the redundancy in natural language.
Preference Pairs Construction To evaluate the effectiveness of short-to-long alignment compared with original long-contex alignment, we construct preference pairs based on both short and long contexts using instruct models. Specifically, for each input context and question,
first generate N = 32 Chain-of-Thouuht responses using a sampling temperature of 0.85 to encourage response diversity. We then apply the sub-em method to identify the chosen and rejected responses within the generated samples. A final preference pair is formed by randomly selecting one from the chosen and the rejected candidates, respectively. Samples where all responses are correct/incorrect are discarded.
To enhance the validity of the comparative analysis, we prioritize constructing training examples from the intersection of samples that can be successfully constructed using both short and long contexts. This ensures that differences in model performance stem primarily from alignment strategies rather than data distribution. The final size of each train set is 5000, as detials shown in Table 2 and an example is shown in Table The prompt template used for data construction is illustrated in Figure 6
Altemaatvve evaluation methods, such as I.I. -ungee, may be employed provided they can differentiate preference pairs.
20

---

Published as a conference paper at ICLR 2026
Table 6: An example preference pair sampled from Qwen2.5-7B-Inss --Istructtt using the short context
(Eshort), Certain reasoning details have been omitted and denoted by " "... for conciseness.
question answer
Nhen was the institute that owned The Collegian founded? 1960
To Find when the institute that owned The Collegian was founded, let's fol these steps:
chosen
Identify the owner of The collegiant....
Find the founding year of Rouston Baptist University: answer
1960
answer the quest' on, need to idertify which institute "The Collegian" is asscciated with and then find its founding date. However, the provided passages not explicitly 1ink The Collegian founding date for the university is given in the passages provided.
rejected
The answer is: No answer.
Write un accurate and concise unswer for the given question using only the provided search results (some of which might be irrelevant). Start with an accurate, engaging, and concise explanation based only on the provided documents. Must end with 'The answer is: {(your final answer) J. Use an unbiased and journatistic tone. If the question cannot be answered, end with 'The answer is: No answer'. Input:
Passage "[TTTLE-]]"
[DOCUMENT-I]
Passage "[TTTLE-2]":
[DOCUMENT-2]
Question: [QUERY] Think step-by-step.
Answer:
Figure 6: Prompt template used for data construction and training, adapted from Li et al. 40
E IMPLEMENTATION DETAILS
In this section, we describe the implementation details of our experiments, including the configurations for model training and evaluation.
E. MODEL TRAINING CONFIGURATION
General training settings We train our model using LLaMAFactory [86] for data with a maximum length < 8K. To enhance training efficiency and better utilize GPU memory, we employ Flash- Attention 2 [13] and DeepSpeed ZeRO stage 3 with offloading (55) strategies. All models are fully trained on four NVIDIA A 100 80GB GPUs in bf16 precision. We use the Adam optimizer [45] together with cosine learning rate scheduler. The warmup ratio is set to 0.1 and the total batch s size 64. For fair comparison, for each method we select the checkpoint that achieves the best performance the QA tasks in LongBench- V1 during single training epoct as the final model to be evaluated across different benchmarks.
Supervised fine-tuning (SFT) For SFT, the maximum learning rate is set to 10
Original preference optimization We apply DPO, SimPO, and ORPO with smaller maximum learning rate 1 x 10-6, Following the original methods [54 for DPO and ORPO, we set the = 0.1, and for SimPO, we set 3 2.0 and 7 1.4.
Short-to-Long preference optimization (SoLo-PO) For SoLo-PO, we maintain the same training parameters in the original method. Specifically, SoLo-DPO, SoLo-SimPO, and SoLo-ORPO achieve optimal performance on LongBenchVI with the reward alignment coefficient in Equation 9 set 3, and respectively, for Qwen2.5-7B, while SoLo-ORPO yields better performance with & = 4 for LLaMA3.1-8B.
Results show that, dataset, all training methods achieve their best performance within single epoch.
21

---

Published as a conference paper at ICLR 2026
LongPO We train Owen2.5-7B-. Ins-tuttt constructed short-to-long dataset with the publicly available LongPO codebase using the default hyperparameter settings, including the optimizer, learning rate and the learning rate scheduler, except for the batch size, which is set 64 to match our setup. In addition, we also evaluate the publicly released model checkpoin for direct comparison.
E.2
EVALUATION DETAILS
E.2.1 EVALUATION BENCHMARKS
To comprehensively evaluate the capabilities of SoLoPO, we conduct experiments across a diverse set of benchmark datasets, follows:
QA tasks from LongBench V1 and RULER Given that our training data is derived from the multi- hop QA dataset MuSiQue (64, with maximum sequence length of 8K tokens, we primarily assess SoLoPO's performance on Long-contesd QA tasks within 32K context size. Specifically, we use the QA tasks from LongBench domains. These include single-document QA tasks such as Qasper [14), Narrative (35] and MultiFieldQAA [ as well as multi-document QA tasks including HotpotQA 78), MuSiQue 64
evaluate the model's generalization across various real-world
and 2WikiMQA [29]. Additionally, incorporate synthetic QA tasks from RULER SquadQA [56].
HotpotQA (78), and MuSiQue [64]-at varying context lengths (4K, 8K, 16K, and 32K tokens) to further analyze the model's length extrapolation abilities.
LongBenchV2 To explore the potential of SoLoPO in more diverse and longer-context scenarios, we further evaluate it on the full suite of tasks LongBenchV2 (6. This benchmark covers wide range of long-cone context tasks, including question answering, abstractive summarization, and in-context learning, with input lengths spanning below 32K words, between 32K and 128K words, and beyond 128K words.
Open LLM Leaderboard Prior works [72 16 [1] note that aligning models for long-conter
tasks may forget their short-context capabilities. To assess short-context performance retention of
different methods, we adopt evaluations from the Open LLM Leaderboard [ 19]. These include
widely used tasks such MMLU-Pro (66], MATH (27) GPQA [57], IFEva 87]. and BBH [60],
which valuate general knowledge, mathematical reasoning, scientific (chemistry, biology, physics) knowledge, instruction following, and complex reasoning, respectively.
NIAH-Plus As described in Section 4.2 to better understand how different training strategies affect the contextual knowledge utilization capability, we employ the NIAH-Plus [84] benchmark. This needle-in- in-a- aredle-in-ahhyystack QA benchmark includes both single-document and multi-document settings, and designed to directly probe a model's capacity for context-aware retrieval and multi-step reasoning.
E.2.2 EVALUATION SETTINGS
Prompts for Evaluation For QA tasks in LongBenchVi and RULER, we use the same prompt template as employed during data construction and model training, which is illustrated in Figure For all other benchmarks mentioned this paper, we adopt their publicly released prompts. Specifically, as shown in Figure 7 for LongBench V2, we employ a single-stage chain-of-though prompting strategy to generate answers directly, differing from the official two-stage evaluation protocol. For the Open LLM Leaderboard and NIAH-Plus benchmarks, we follow the default prompts used in the and NIAHaystack-PPl PLUS repositories).
official implementation code (Im-evaluation-harnt
Decoding hyperparameters We greedy decoding for evaluation LongBench RULER, and NIAH-Plus. For other benchmarks, we follow the official decoding settings with temperature
ht http::///thhob.ooo//ooooo-----s0
leaderboard
22

---

Published as a conference paper at ICLR 2026
Please read the following text and answer the question below.
<text>>
[CONTEXT]
</text>
What is the correct answer this question: [QUESTION]
Choices:
(A) [CHOICE A]
CHOICE B
CHOICE C
(CHOICE_D) D)
Let's think step by step. And format your final choice follows: "The correct answer is (insert answer here)".
Figure Prompt template used for LongBenchV2 evaluation, adapted from Bai et al. 66
values of 0.1 and 0 for LongBenchV2 and the Open LLM Leaderboard, respectively. analysis for Vongbench is provided in Appendix F
The error
F STANDARD DEVIATION OF LONGBENCHV2
Qwen2.5:78 5-78-lstrutt (wr YARN)
38
Lima3 1-8R-Instrutt
36
34
32
28
29.3
30
25.5
29.7
24.6
29.3
22
24
26
32,5
31.8
33.7
32.1
20
Instuct
Long SFT
Shome DPO
Long-DPO
DDOo
Short SimPO
LongSiimP
SoL--SimnO
Shor-CRPO
Long ORPO
SoLo-ORPO
hnmntet
Shon SFT
Long SFT
Shon ORPO
Long ORPO
SaLo-ORrO
Shop
Figure 8: Overall Performance on Longbench V2. 1. We report the average score with standard deviation across 5 evaluation runs for each model. 2. All of these metrics are reasonable.
As mentioned in Appendix E.2 we use greedy decoding (temperature 0) for all benchmarks except Longbench V2, where the temperature is set to 0.1. Here, we report the standard deviation of LongbenchV2 in Figure 8 and Table where all of these metrics are reasonable.
G MORE DETAILED BENCHMARK RESULTS
We present more detailed results for our experiments presented in Section
Detailed results QA tasks from Longbench V1 and RULER Since the training dataset use, MuSiQue, is designed for multi-hop QA tasks, in Section we primarily evaluate various methods on the QA tasks from LongBenchV/ and RULER. This allows to examine model performance on real-world scenarios and across varying input lengths. Results are presented in Table 7 and Table 8
More detailed results on NIAH-Plus In Section 4.2 to validate whether the decoupled approach may more effectively enhance the model's ability to locate contextual knowledge. we evaluate Expand-Long-PPl (a non-decoupled approach) and SoLoPO (our decoupled approach) on NIAH-Plus with full results presented in T SoLoPO consistently outperforms Expand-Long-PPe across all evaluation scenarios and preference optimization algorithms, demonstrating the effectiveness of our decoupling-based short-to-long preference optimization approach. Figure presents presentt heatmaps of the performance gains of various ORPO over Qwen2.5-7B. SoLo- LOLOO, powered by the SoLoPO's decoupling strategy, surpasses Cong-ORPO and Expand- ORPO NIAH-Plus. It achieves superior information retrieval and utilization across various depths and context lengths in both single-hop and multi-hop settings, fundamentally improving contextual knowledge localization.
23

---

Published as a conference paper at ICLR 2026
Table Detailed Results of QA tasks from LongBenchVI
Single-Doc QA
Multi-Doc QA
Model
AVG.
NarrativeQA
Qasper
MultiFieldQA-En
Avg.
HotpotQA
2WikiMQA
MuSiQue
Avg.
Qwen2.5-77B -nn-nntuett
Instruct
15.8
31.3
41.0
39.6
29.7
Mas
13.4
34.0
39.3
28.9
47.3
63.3
34.5
48.4
38.6
M
21.3
39.4
43.7
34.8
55.4
65.7
46.5
55.8
45.3
17.2
41.3
43.3
34.6
68.0
36.4
5188
43.2
Mas
22.6
41.1
43.4
35.7
60.2
48.0
58.2
46.9
26.1
43.0
44.8
38.0
58.8
50.7
57.6
47.8
M7 Mane
20.3
41.4
42.3
34.7
35.5
67.1
38.1
53.6
24.I
M
16.8
41.0
44.8
34.2
56.3
64.9
43.6
54.9
44.6
Mila
25.5
42.8
46.0
38.1
61.5
68.4
46.0
58.6
48.4
Maa
13.4
34.0
39.3
28.9
34.5
38.6
Marr
21.3
39.4
43.7
34.8
55.4
65.7
46.5
55.8
45.3
24.9
41.4
46.6
37.6
64.3
71.7
48.1
61.4
49.5
L. 1-8--111-----ute
Instruct
12.3
37.3
41.5
30.4
54.8
59.8
33.4
49.3
39.8
Mar
19.1
37.1
42.9
33.0
59.0
65.4
44.4
56.3
44.6
Ma
20.3
39.6
45.1
35.0
56.6
47.1
57.3
M
17.1
38.8
43.3
33.1
55.4
67.1
42.7
55.1
44.1
Marr
19.6
40.1
46.7
35.4
57.2
68.3
40.8
55.4
45.4
21.5
40.0
44.2
35.2
54.8
69.0
48.2
$7.3
46.3
Table 8: Detailed Results of QA tasks from RULER
SquadQA(OOD)
HotpotQA(OOD)
MuSiQue (InDomain)
Model
AVG.
4k
8k
16k
32k
4k.
8k
16k
32k
4k
8k
16k
32k
Qwen2.5-71 10ve2.5-711-Ma 7
Instruct
61.3
49.5
37.0
35.2
45.8
59.6
61.4
51.3
47.7
55.0)
40.9
39.4
24.6
20.8
31.4
44.0
M
72.3
57.5
41.4
49.4
68.2
66.0
44.8
57.9
50.8
46.5
33.0
24.2
38.6
M
70.9
64.4
61.8
54.1
62.8
71.0
67.7
68.4
68.5
55.9
52.1
44.9
Mas
76.2
64.0
41.9
447.8
37.5
74.4
72.2
38.9
59.7
'6.33
62.
537T
"35.0"
33.1
46.0
56.6
M
75.6
66.7
62.5
52.7
64.4
74.6
69.8
70.9
67.0
70.6
62.7
56.3
48.5
39.8
51.8
62.2
70.7
64.4
65.5
70.5
73.2
71.5
67.3
70.6
58.2
55.8
51.7
44.0
52.4
Mas
75.8
62.7
39.4
50.0
57.0
74.8
770.1
33.8
62.6
63.3
61.7
54.9
33.2
33.8
45.9
36.0
Ma
74.0
64.0
45.0
46.4
57.4
74.1
73.8
62.7
61.1
67.9
61.2
54.7
39.6
34.0
47.4
57.5
75.2
67.6
66.9
60.6
67.6
74.3
73.9
71.8
69.9
72.5
58.1
56.5
49.3
42.9
51.7
63.9
Mase
75.2
73.3
69.8
59.0
52.1
40.8
Maseo
70.5
61.2
48.4
45.9
56.5
68.7
69.0
63.4
59.5
65.2
55.4
49.5
38.1
31.4
43.6
55.1
74.8
67.8
65.6
59.2
66.9
73.3
74.8
70.7
67.2
71.5
64.2
62.4
55.7
45.6
57.0
65.1
LLama3.1
Instruct
67.4
53.8
44.9
41.7
52.0
68.1
61.4
$7.3
49.9
59.2
39.4
32.5
26.3
15.1
28.3
M
70.7
66.0
61.7
55.4
63.5
67.6
67.8
66.2
62.5
66.0
56.7
49.1
47.5
Mor
69.4
63.6
64.6
58.9
64.1
66.6
64.6
63.7
62.0
64.2
55.0
48.9
44.3
39.5
46.9
58.4
Ma
20.0
65.5
67.7
522.0
63.8
61.9
66.
65.6
63.0
63.3
34.5
31.
43.7
36.2
36.0
38.4"
M
68.5
65.1
64.9
59.2
64.4
68.5
66.7
66.7
65.8
52.7
48.5
45.1
38.9
Mal
67.0
63.5
64.5
59.8
63.7
70.3
67.0
66.0
66.8
67.5
55.8
57.1
49.8
48.1
52.7
61.3
H MORE EXPERIMENTAL ANALYSIS
H.1 MORE ANALYSIS ABOUT SOLO-DPO/SIMPO ON LONGBENCHV2
Table presents results of Qwen2.5-7B (w/ YARN (51) on LongBench For Swve2.5-7BB (w/ yarN), SoLoPO consistently outperforms original PO methods across all evaluation context lengths and difficulty levels. Specifically, (1) SoLo-ORPO also surpasses vanilla PO across all dimensions:
(2) SoLo-DPO achieves the best overall performance, particularly on contexts 2 32 and hard samples, likely due to the reference model ensuring better generalization; (3) SoLo-SimPO shows relatively weaker performance, possibly due its reward model relies on normalized prediction probabilities, which underperform long-context evaluations like perplexity observed by Fang al. (18)
24

---

Published as a conference paper at ICLR 2026
Table Comparative performance of SoLoPO and Expand-Long-PO on NIAH-Plus (within 128K context size). As shown in (f), SoLoPO consistently outperforms Expand-Long-PO validating our decoupled short-to-long preference optimization approach.
Model
Single-document QA
Multi-document QA
AVG.
Qwen-2.5-78-Instruus
Instruct
35.66
52.63
44. 14
M
55.98
68.02
62.00
Mas
59.35
3.37)
71.76
3.74)
65.56 3.56)
M Saa
31.81
53.61
52.71
M
60.85
9.04)
72.05 (1 18.44)
66.45 (1 13.74)
M Moan
59.60
69.92
64.76
61.64
2T04)
71.46 1.54)
66.55 (1 1.79)
-Lama33--B--nstruuc
Instruct
32.04
32.8
32.42
Mmc
43.69
42.57
43..33"
M M
43.86
(1 0.17
52.96 (1 10.39)
48.41 (1 (.28))
LapO-ORPO
Expand-Lang-ORPo
SOLO
Purcont
Tegith Pecoent
23.94%
sale
25.90%
2002
Tokon Linit
(a) Single-document QA setting.
CIALCOONS
6.93%
17.29%
18.83%
(D monoo
with
Token Limil
Token Limil
Token Limil
(b) dulti-document QA setting.
Figure 9: Comparison of performance improvements achieved by various ORPO methods relative to and SoLoPO demonstrate significantly
Qwen2.5-7B on the NIAH-Plus [84). Expand-Long-On greater improvements than Long-ORPO highlighting the effectiveness of short-to-long alignment. Moreover, SoLoPO provides greater gains than Expand-Long-ORRe ORPOO validating the effectiveness of decoupled approach.
H.2 TRAINING DYNAMICS DIFFERENT SoLo-R APPROACHES (chosen-only VS. both)
As demonstrated in Figure T0a the margin curves of the both RoLo-RA consistently lie beneath those of the chosen-only approach, with its ultimately converged margin values being substantially lower. This observation indicates that the chosen-only BoLo-RA exhibits superior fitting capability along the Stong dimension. reveals that during the initial optimization phase, the both SoLo-RA induces an anomalous transient increase in the probability of Po(y) #longg) which should theoretically follow monotonic decreasing trend. This suboptimal convergence behavior ultimately leads to significantly inferior optimization outcomes compared to the chosen-only approach as shown Figure 2a
H.3 ABLATION EXPERIMENTS ON SoLo-RA
25

---

Published as a conference paper at ICLR 2026
rewards/log margins
logps/long rejected
0.07
Normalized Valuess
0.06
Both
Normalized Values
-0.50
Both
Chosen
Chosen
0.05
-0.55
0.04
-0.60
0.03
-0.65
0.02
0.01
-0.70
20
30
40
60
70
10
30
50
60
70
train/global step
train/global step
(a) Torpo (Tlongs yw) 7 Torpo (Ttongs yt).
(b) Log prob. of ps(1)) (tonny.
Figure 10: Changes of reward margins and log prob. of rejected response during SoLo-ORPO training given long contexts. Compared to the both approach, chosen-only short-to-long reward alignment achieves (a) larger reward margins, ensuring higher reward accuracy, while (b) simultancously apply- ing sufficient penalties rejected responses, reducing their prediction probability. The chosen-only approach exhibits more precise reward modeling, ultimately achieving better alignment outcomes.
In Section 2.2. we hypothesize that SoLo-RA en-
Table 10: Ablation studies for SoLo-RA on NIAH-Plus (within 128K context size).
hances the model's contextual knowledge localiza- tion ability (see Appendix [.9 for detailed discus- sion). this section, we conduct ablation studies on NIAH-Plus to further validate this hypothesis.
Model
S-Doc QA
M-Doc QA
AVG.
Qwen-2.5-7B-Inster
To conduct the ablation analysis of Sol-o-R,, we focus on the performance of models trained with Short-PO, Expand-PO, and SoLoPO on NIAH-Plus, for the following reasons:
Instruct
35.66
52.63
44.14
Mas
51.25
64.70
57.98
M Mare
55.98
68.02
62.00
crasser
M
59.35
71.76
65.56
Maso
50.84
63.59
57.22
Removing RA from SoLoPO's optimiza- tion objective yields short-PO;; however, this also eliminates the Zlong from the training data, making a direct comparison unable to disentan- gle the gains from data length generaliza- tion [90].
M Maro
51.81
53.61
52.71
M Paski
60.85
72.05
66.45
M MORO
45.02
59.17
52.10
M FORPO
$9.60
69.92
64.76
61.64
71.46
66.55
LLama3.1-8BIInstee
Expand-PO and SoLoPO trained iden- tical data except for the absence of Tshort in Expand-PO, allowing their comparison to mini- mize potential confounding effects from training data length.
Instruct
32.04
32.80
32.42
M M
37.20
"37.77
37.49
M couro
43.69
42.57
43.13
M Morro
43.86
52.96
48.41
As shown in Table 10 SoLo-PO consistently outperforms both Short-PO and Expand-PO in contextual across different PO algorithms and base
knowledge localization tasks with 128K context length models. These results provide further evidence that SoLo-RA more effectively strengthens the capacity of the model to localize contextual knowledge.
H.4 IMPACT OF REWARD ALIGNMENT COEFFICIENT a IN SoLo-DPO AND SOLO-SIMPO
Figure shows the performance of SoLo-DPO and SoLo-SimPO under varying in the Qwen2.5- 7B setting. The results indicate that the optimal a values are 3 for SoLo-DPO and for SoLo-SimPO, and in most SoLoPO outperforms Long-PO, suggesting its stability across different values.
H.5 EFFICIENCY ANALYSIS OF SoLoPO
Thanks to the reduced overhead handling long texts, SoLoPO offers notable advantages over the original algorithms in both computational efficiency and memory usage. In this section,
26

---

Published as a conference paper at ICLR 2026
Avrenge Performince on QAA-- (DAs-Longheneh) L.nnghenchV
48.4
48.4
47.9
47.8
46.7
47.2
47.5
47.0
SoLo-DPO
Sol.o-SimPO
Long-DPO DPO
Long-SimPO
42
0.5
10
Reward Alignment Coefficient
Figure 11: Performance with different a in SoLo-DPO and SoLo-SimPO the Qwen2.5-7B setting. The optimal values of a for SoLo-DPO and SoLo-SimPO are 3 and respectively.
present detailed empirical comparisons of runtime efficiency and provide theoretical analysis of the computational speedup.
Training Implementation Details The experimental setup for Figure lb adheres to the same training configuration described in Section E.I For experiments Section 4.3 we employ various
optimization strategies of ZeRO [55] to maximize GPU memory utilization while enabling training on longer sequences. The specific training configurations are detailed in Table
Table 1I: Implementation details and run time SoLo-ORPO and vanilla ORPO under varying
lengths of Tiong. 1. the default configuration is Zero stage 3 without offloading 2. "2-Stage Forward" indicates sequential forward passes for (yw; Flong) and (Hl, Zlong), as opposed to the default concatenated forward strategy. 3. "OOM" denotes CUDA Out-of-Memory errors. 4. SoLoPO significantly improves the training efficiency of the vanilla ORPO.
Length
Method
Offloading
2-Stage Forward Wtong
Runtime (min)
1K
vanilla
54.00
vanílla
72.32
4K
SoLo
66.63 (1 8%)
vanilla
145.11
8K
SoLo
109,42 25%)
SoLo
83.62 (1 42%)
vanilla
235.98
12K
SoLo
144.21 39%)
vanilla
OOM
16K
SoLo
179.20
vanilla
Õ0M
19K
SoLo
205.98
SoLoPO When the lengths of Ztong ranging from 4K to 16K tokens, we employ a two-stage forward mechanism within LLaMAFactory [86] to perform SoLoPO training. Specifically, for the short-context PO, we apply the concatenated__orwall ffncttoonnn function directly on (Yw- Tshort) and (yt, Jshort) to obtain log To [sshort) and log тe (y) Tshorr). Subsequently, we conduct separate forward pass over (Mw 2long) to compute log (yr Zlong). Finally, the SoLoPO loss is calculated based on the corresponding loss function Table T
Vanilla PO When the lengths of Ftong is less than or equal to 8K tokens, (Vw #long) and (yt, Fiong) can be efficiently processed using the concatenated. for.ff.ooarddd function, allowing for straightforward loss computation. However, at &tong lengths of 12K tokens, the use of leads to out-of-memory (OOM) errors. Thus, we adopt 2-stage forward upproach-processine processing (yw: Elong) and (yt, Zlong) sequentially. For sequences as long as 16K tokens, even this serialized method becomes infeasible. necessitating the use of sequence parallelism techniques to enable training. For even longer (Tlony) exceeding 16K tokens, only a
concatenatedd_orwee
ama 1 factory/
train//pootrraierr.po
27

---

Published as a conference paper at ICLR 2026
sequence parallelism [33 training strategy becomes feasible. While these alternative approaches mitigate memory constraints, they inevitably increase overall training time.
Computation speedup analysis of SoLoPO Following Bai al. (2) define the compression rate p E (0. 1] as the ratio of the length of Eshort to that Tlong Specifically, if the length of Glong is denoted by N, then the length of shorrt is given by p.V. We focus on the computation incurred by the policy model during training and ignore reference models. Since Transformer operations scale quadratically with sequence length (FLOPs n²2,, the total training computation for vanilla PO and SoLoPO can be expressed as:
FLOPspo 2N2, FLOPSSoLOPO (2p² + 1)N2.
(11)
Consequently, the computation speedup ratio of SoLoPO over vanilla PO can be expressed as:
FLOPsro
2
Speedup(c) = FLOPSSSaoro 2p +1 P PE (0,
(12)
This indicates that SoLoPO achieves computational efficiency gains when p when Zshort is less than approximately 70.7% of the length of Zlong.
1 v2
0.707, i.e.,
Potential Approaches for Optimizing Training Efficiency on High Compression Rate Tasks For tasks with high compression ratios, such as long-context machine translation ( p=100%) the linguistic redundancy hypothesis underlying SoLoPO no longer holds. In this scenario, SoLoPO degrades to vanilla PO and offers no training efficiency gains. noteworthy that standard attention mechanisms in LLMs exhibit redundancy, which enables the application of KV compression 44 or sparse attention [71] techniques. We hypothesize that this model-inherent redundancy can be leveraged for representational compression (e.g., gist-token (22 Consequently, for tasks lacking linguistic redundancy, the short-to-song paradigm could potentially be applied based on the model- inherent redundancy, thereby reducing training time and memory consumption for long-context scenarios.
Table 12: Performance of Qwenn.5-Instruct-145 trained with different methods on Longbench V1 QAs and RULER-QA -As. Across benchmarks. SoLoPO consistently outperforms vanilla PO algorithms.
QAs- BAs-oongBenchVi
QAs-RULER
Run Time
Model
S-Doe QA
M-Doc QA
Avg.
4k
8k
16k
32k
Avg.
/min(()
72B Instructt
37.80
61.10
49.40
65.70
64.4
61.20
55.0
61.60
14B-tnstt
34.40
52.07
43.30
56.60
52.27
49.69
43.86
50.60
Ma
36.20
59.13
47.70
64.47
59.46
55.67
46.31
56.48
Mar
34.53
61.07
47.80
65.72
61.44
52.56
43.83
55.89
Maoo
37.83
65.33
51.60
71.32
65.79
63.12
55.97
64.05
Mos
36.63
66.63
51.60
73.37
69.49
67.69
59.83
67.60
249
Muo
37.80
67.20
53.40
72.95
70.53
68.52
62.03
68.51
172
Masro
37.47
64.30
50.90
71.43
65.87
63.21
55.69
64.05
M
36.77
66.10
51.40
70.33
66.24
63.27
56.91
64.19
248
Misro
38.40
65.67
52.00
71.79
67.85
66.52
60.47
66.66
169
Maro
39.37
63.80
51.60
68.63
63.85
61.47
52.96
61.73
M
38,80
62.73
50.80
68.69
64.19
62.16
52.83
61.97
248
Moño
39.80
65.70
52.80
72.45
67.85
65.10
60.78
66.54
170
H.6 ON THE SCALABILITY OF SoLoPO TO LARGER MODELS
To examine the scalability of SoLoPO to larger models, we conduct experiments on Qwen2.5- Instruct-14B with different preference optimization methods, constrained by available computational resources. The evaluation is performed on the primary benchmarks used in this paper, LongbenchVI- QAs and RULER -As. The training and evaluation settings follow those of 1iven2.5-Insrruct-77 except that the parameter in SoLoPO is set to and experiments LLaMA-Face -aat-Factryyy (0. 9. 9.1). As shown in Table [2] SoLoPO consistently outperforms the original PO algorithms across different benchmarks while significantly improving training efficiency.
on 8x H20 GPUs with
28

---

Published as a conference paper at ICLR 2026
I THEORETICAL DERIVATION AND SUPPORTING ANALYSIS OF SOLOPO
In this section, we formulate the theoretical foundation for the short-to-long preference optimization (SoLoPO) method through novel reward loss function, and develop a distance metric condition for applying the framework for generalized distance metrics. Furthermore, we systematically extend this framework to mainstream preference optimization paradigms, including Direct Preference Optimiza- tion (DPO), Simple Preference Optimization (SimPO) and so on, demonstrating its methodological generality.
1.1 PROOF OF LEMMA
Lemma 1, If the function f(-) of equation
4 is convex, then the following inequality holds true:
1 (Ttong Flongi ywe U1) S B3,. (Flongs Eshort' Ywu Vw:)
(13)
+ 13n.g (Ishort; Uishorti Yhes y1) + 13n. (Eshort, Flong' Ut 91)
Proof.
(n, Flong's Yw y1)
= f(n (Zlong: yw) r Flong 91) - 71)
f ( (Toong yw) shorts Fw!
+ To (Tshort: (w r shortt yho)
(Fshort (w) r (Flong: 111 71)
ff(n [ + D2 + Ag - 7
1
-(G1" (3A) + 3A2 7 +34 -)]]
[f (3A1 - 7)) + f(n (3A2 7)) + f(n (3A3 7))] by Jensen's Inequality
3 Ilan.3 3 (Tlangs 3 Sshorti yhos yww + 13n,3 (Eshort: Ushorts Hwe 31 + 1an. (Ishort, Glong; D1 91)
where
A To (Tlong Fw: - T (Eshorts F/)
A T ahortt yho) T shorts M1)
Ag (Zshort 31) - T Too (Ilong y1)
I.2 PROOF OF THEOREM
I
Proof. Directly applying expectation Ex, -D, to inequality 13 and using assumption we Vw Fiong
can obtain the following inequality:
E (D Driong: D Dziang Dy D Wionny D yo in (stom))
L (Driong: D Dessor'' D Mool sooor D (astore)
(14)
Lzn,3 (D Drshhree D Dashore's D laaner D You wahort)
Lsn,3 (D D Darono's Dyn aamonts D lesnore))
29

---

Published as a conference paper at ICLR 2026
We prove only the second term of inequality 14 the proofs for the remaining terms follow in the same manner.
E,, DD. Nw-201 - Un > (rtomy sn, 3 (Fshorts Eshorti Mues 3/)]
(15)
EEg DD. vn D D2 (P yt Flong Man. 3 (Ishorts Eshorts yw y1]
(16)
P(yo > yt (iiog)
-Eg DD. We v2 ~ P(y > 311 ssort)
P(y > (n([short Man. (Ishort Der Vt)
(17)
SEg De. Ww Dz (yvo yt shorrt N3n. & (Eshorti Ywe 31)
(18)
=L3n.3 3 (Dushore
Dr Dashors? D tre vn woohore Dy yi wanore)
(19)
Now, we prove the inequality 7 We only need to consider the sum the first term and the third term:
3n 1 (Dotono' Do Dashort D Mull D (oxe + 3 (Dashorer Dasonu? D yi leshare D yi (E
(20)
=E 2. 9m. Dy
[P > yel wehore O3,, 3 (Tlong: Eshorti Vwr Vio) lan 3 (Fishorts Plong' Ut, vt))
(21)
E ~i. Dr
1n... (Tlongs Fshorti Mw yw)) + 130.3 (I (Ishorts Vlongi Mes 311
(22)
E 2.. :
[5(1) (3 (1hn) - 7) + f(n (34x(11) 7))1
(23)
E I.D.D : [f(n (3 - 7)) + f (3Aa (mn) 7))1
(24)
D D2
Ex DD. w (1 (1) - -)) + f(n (3Ax(v) - 7))1
(25)
E. Dr. w [1 (3A) (v) - -7)) + /(n (-3A (1) - -))] - Ex DD y- D. |/ (3 - 17)) + 1(-3nA (v) m))]
(26)
(27)
<En De D. ((3n (ro (Ishort y) r (olong y))
(28)
where s-)) satisfies f(z + 7) + ( - s(121) and
(v r (Tlong 4) T (hhortt 9) A3(y) = r r (Fehort,)) v) - ro (Foogg y)
(29)
(30)
The introduction of s(-) is to unify the two terms, To (Ztong y) and I's (Z y). into a single expression. This unified expression serves quantify the distance between (Ttong y) and ro(2 rs(Tshort, y). Building upon this foundation, we further generalized this concept in Appendix I.5 leading more generalized theorem, which may provide valuable insights and inspire future research directions. Given that s(-) serves as an upper bound, tighter instantiation is theoretically preferred; we provide empirical evidence for this claim in Appendix I.8
1.3 SoLoPO FOR f(x) x²2
Proposition 1. Following the notation of Theorem if we take f(x) x.2 the inequality becomes:
1 f (Fshort) + +37² Er DD. D, Ir (Ashort: y) - ro (diong, f) + 7? (31)
2
(n 5 3
Proof.
f(x + 7) f(=x((++)) (x 7)2 + 2x² + +22
Therefore, by using the Theorem we prove this proposition.
30

---

Published as a conference paper at ICLR 2026
1.4 SoLoPO FOR f(x) log (())
Proposition 2. Following the notation of Theorem if we take f(x) = log (ar), then the inequality can be:
2
(2no())oogg) S f3n (Ishort) + 1f. Piz. DD.. wyD, (T y) ro (aloooo v) log(1 + 27
(32)
Proof.
3. 3 (D (Deingg Dasnors's D Pahore D Muul shoor) + Lan (Dzenors: Deiong? D 2n sinott D
Me it sshhort
(33)
<B n. B.-DD D Man j (Zlong: Lshorti v. 1an (Eshorti Flong's y y)]
(34)
Nw Mi
(35)
For brevity, we omit the expectation in the following derivation.
1 (Mang Tshorti yno) ) + 13n (Eshorts Mlnng: y 2)
(36)
=f(3n (ra(Iiong: y) r @chorts 8)) 17 -7-7) f(n (ro(2short, v) -r v)) 77)
(37)
we denote To (Flong: y) (Tihhrtt:
( y) as TI, Tss
(38)
=f(3n) (ri ra) 1) + [(3]) (r. -ri) 77)
(39)
log (3n) T! 3r Ts - 17 7) - llog (3n re
3n Ti Ti- -77777)
(40)
1
log
log
1 exp (-(3m) Ti 31/ rs 11- 7)}
1 + exp (-(3n To
31 ri 77 7)}
(41)
e .
3n.
log
3. + 0 log Bp + 2
(42)
= 31 ri 37 re + l + (dm)) + log(e 2
(43)
3 DD.. Im ral + 21gg(1 + edy)
(44)
Dr
In the following, we prove the inequality (a).
If r I Fs, then
3n ri 37. r log( ( Sp + T. log(esnr (0
(45)
< 33 r1 - 31 T. + 30 + 2 ) + log(e
(46)
-3n 3m + 61/ r. + 2 log(1 (1 + en7
(47)
3r) T. - 3m Ti + log + )
(48)
By symmetry, we can easily obtain that if r. < rı, then
-3 3 torr..tyyy00 + llg 3n (31--11) < 3n-r: +2 (
(49)
Therefore,
-3n-ri- -n--1-33-- r. -an-rr++llgge
- < 33n b 1 2 (
(50)
I.5 GENERALIZATION OF THEOREM
Theorem 2. Let Dp(21,02) (By~D, r0(22, v) any divergence D(11522)
satisfies
(212 S C1 (2)1)
(51)
31

---

Published as a conference paper at ICLR 2026
where C1 are positive constants, then the following inequality holds true for the convex function f(z)= =
log (I), as in the settings of DPO, SimPO, and ORPO:
L (Ptong: Diongi D Dylsions' Dy Pylstona)
(52)
3 (Dehorts Dahorti D Pylzahorrs Dyt Pylzanore) 1] Cj DD. (I Tiong)) )1 + C2 (53)
where C2 = 2/ (log(1 + +))7
Theorem 2 guarantees that any new distance satisfying the inequality 51 can substitute the absolute distance.
Proof. This theorem can be proved by directly using the proposition 2
1.6
THE GENERAL FORMULA OF SoLoPO Loss FUNCTION
The general formula of Short-to-Long Preference Optimization (SoLoPO) loss function:
LSoLoPO
E 2~ a~Dassore Vw - DD Dii
[f(n] |To( yw) (w) To( 21) - -]]]
+ a Eg. DD WDD, ( ahort. y) (Tiong v)()
where f a convex function, and f(x + + f(-x 7) S s((z]) for some function s. 7, o. are hyperparameters.
1.7 EMPIRICAL EVIDENCE FOR ASSUMPTION
In this section, we verify Assumption
since Tlong contains more cask-irrelevant information than Tshort making harder for the model to distinguish preferences when conditioned on Zlong:
p(yno > yr [1ong) S p(yo yn ((lootort)
(54)
where plu > |x) = (xr (x, Bw) - r" (xI r" denotes the optimal reward model, and a is
the sigmoid function.
In the absence of an optimal reward model, we employ a model Tinal trained via DPO to estimate the reward margin for (yw vn following the reward computation formulation defined in DPO:
ffnal x) x)
2T final (y x))
TDPO - rppoo
log Tin
3 log
(55)
Trof (y
x))
Specifically, we adopt Owen2.5-7B. 5-7BB-Inttructt as the reference policy Tel and perform DPO training on data with a context length of 1K to obtain the final policy Tfina- We set the short-context length to 4K, from which we sample preference pairs you > Y/ ~ Teer(y Wshoon,, and subsequently expand them to lengths ranging from SK to 32K to obtain Tongg This design aims to emulate reward model with a non-trivial scoring capacity that is nonetheless susceptible to noise, in order to meet the preconditions of our assumption. For each length, compute the proportion satisfying Eq. 54 based on Eq. 55
Table 13: Proportion of cases in which Assumption 54 holds for preference pairs (yw > 31) sampled from Eshort (length 4K) and evaluated on Zlong with varying lengths (8K-32K), using model trained via DPO on 1K context length. Longer contexts introduce additional task fask-inrelevant information, leading to a gradual increase the satisfaction rate and stabilizing at approximately 95%
Length of Tlong
8K
12K
16K
20K
21K
28K
32K
Hypothesis Validity Proportion
80.58
86.41%
91.26%
90.29%
96. 12%
95.
94.17%
32

---

Published as a conference paper at ICLR 2026
Results shown in Table 13 As the context length increases, the amount of task-irrelevant information grows, and the proportion satisfying the hypothesis gradually rises, stabilizing at around 95%. Considering potential inaccuracies in reward estimation, such consistently high proportions provide substantial support for Assumption 1
1.8 THE IMPACT OF THE TIGHTNESS OF s()) SoLoPO's PERFORMANCE
Theoretically, tighter corresponds to stronger upper bound, which may improve empirical performance. contrast, a loose upper bound often results in vague optimization directions and may cause the optimization to converge to a local optimum. For example, consider minimizing x² ---22 = (1² 2), whose global minima are at a = 1. we use an upper bound x², the global minimum of this upper bound is at 0. Therefore, if the upper bound is too loose, the obtained solution may deviate from the true optimum.
x²
We compare tighter setting s, (2) |ar] (used in the paper) with looser setting s1 | (1) + 2 se (2) on Twen2.5-7BBInnsrueee while keeping all other training configurations identical. As shown in the table below, the tighter s(-) generally achieves better performance, except in the MD-QA scenario of LongBenchVI where SoLo-DPO(s.) is slightly worse than SoLo-DPO(st). Since 9, S in our experiments, can infer that looser s(-) tends to yield worse performance.
Table 14: Impact of s-) tightness on performance. Tighter s(-) generally improves performance.
QAs- LAs-LongBenchVi
QAS-RULER
Model
S-Doc QA
M-Doc QA
Avg.
4k
8k
16k
32k
Avg.
MDPO
35.3
57.8
46.5
64.2
62.6
61.2
57.4
61.4
SoLo(ai)
M DPO
38.0
57.6
47.8
66.4
64.5
62.7
57.7
62.8
SoLo(s.)
Po SoLo(si)
36.8
56.7
46.7
67.8
65.0
61.9
57.2
63.0
SimPO SoLo(s.)
38.1
58.6
48.4
69.2
66.0
62.7
57.8
63.9
SoLotai) MORPO
36.1
57.8
46.9
68.1
65.4
60.5
56.2
62.6
37.6
61.4
49.5
70.8
68.3
64.0
57.3
65.1
SoLo(s:)
1.9 DISCUSSION ON THE MODELING OF SoLOPO
a. Two key abilities in long-context scenarios Unlike short-contexi tasks such as mathematics 33), which can directly leverage the model's inherent (contextual knowledge) reasoning ability, long- context sack such as question answering (64] or information extraction 773 also require the model possess contextual knowledge localization skills, i.e.. the ability to identify task-relevant information Cret from a long context Stong while ignoring irrelevant content Cir- In other words, the model needs to first identify the key task-relevant information Grel from the context Clong and subsequently perform reasoning upon it (38).
Explicit modeling of two key abilities in SoLoPO Recall that the optimization objective of SoLoPO is defined as follows:
LSoLoPO
EawD Droort Bhe wD Ye i1 (yllesborr)
f 33 (no vw) ra 31)
7 3
(56)
shout content preterence oprimization
+a B. DD 12 D ( (ww) Su Wi To (snoes)
[3(3 ( (IThortt y) To (Xloogg v)
(57)
short to long reward alignmens
SoLoPO explicitly models the two abilities in decoupled manner:
Contextual knowledge reasoning: Since Ishort consists of the task instruction I and the task- relevant content Crei, iee. Eshort == [Creti 4]. SoLoPO directly enhances the model's ' inherent reasoning ability via short-context preference optimization (Eq. (56)).
33

---

Published as a conference paper at ICLR 2026
Contextual knowledge localization: SoLo-RA (Eq. 57 encourages the reward model rs to implicitly predict Eshort ~ ((Zshort [Zlong) by minimizing the divergence between Pshort == [Cret; I] and the ground-truth short (rrll I]. In preference optimization, where the reward model re and the policy model Ty coincide, this process also strengthens the policy model's ability locate relevant knowledge Gret for task I within long context Clong-
Taking SimPO an example, where s((zc] = |a]] + C and the reward defined ro v) B log (v)) For brevity, we set 1l = and omit constant C, the SoL.o-RA loss becomes: [y]
SoLo- RAsimPO E 2.D. Dy yy (Mllshhksshrr))
log ( [shert)) log To [[oog))]
(58)
which encourages тe to produce an output with the same likelihood whether it is conditioned on Thhort or on the full input Flong. Under this objective, SoLo-RA implicitly guides the model to extract from Zlong only the minimal sufficient information needed to behave as if conditioned on shortt That is, enforces:
smplicitly
To ( Tlong) x Te (y Tshort)
go (Ilong) Zshort
(59)
where go' can be interpreted presses Mong into representation functionally equivalent to Ishort-
internal attention mechanism latent projection that
This learned behavior aligns with the principle of contextual knowledge localization.
I,10
SUPPORTING ANALYSIS FOR CHOSEN-ONLY SoLo-RA
1.10.1
THE ORIGINAL OBJECTIVE SoLoPO THEORETICAL AND EMPIRICAL SUPPORTED
From an theoretical perspective, LoooPO decomposes long-context preference optimization (PO) into short-context PO and short-to-long reward alignment (SoLo-RA) ($ 2.2 The experimental analysis in Section 4.2 and Figure 2a shows that directly using the SoLoPO objective abjective-applying SoLo-RA jointly to both the chosen and rejected responses (both So)o-RA) achieves superior performance to Long-PO across most settings. These results offer both theoretical proof and empirical validation for the original optimization objective of SoLoPO with both SoLo-RA, as defined in Eq. (57)
1.10.2 SUPPORTING ANALYSIS FOR CHOSEN-ONLY SoLo-RA
In practical scenarios, we further consider variant, SoLoPO with chosen-only SoLo-RA, which is motivated by two factors:
1. Y1 may not always exploit the key information in the context. For example, y/ might simply respond. "No task-relevant content can be found in the context, so the question cannot be answered." In such cases, enforcing SoLo-RA may not improve, and could even degrade, the model's ability to locate or reason over relevant long-context information.
2. We aim to reduce the amount of long-text processing during training, thereby improving training efficiency.
In addition, we further examine the theoretical validity of the first motivation from a data-sampling perspective.
Relation aT Ishortt 2 ( Flong) typically holds owing to the practical data sampling strategy. Recall that preference pairs in SoLoPO are obtained based on short contexts, as defined in Eq. (5
Yo > Yt ~ (y
(60)
For arbitrary Eshort and Flongs the Kullback-Leibler divergence 68 satisfies:
/
|| (zg((tong)) (ryaorrs
[log ग ( (H)(aaaor)) - log ( ( (a/or)))))) dy 2 0.
(61)
This implies that, when sampling y ~ Te(y) Tehorts) the quantity log T Iehort) -short)-logg log T Tlong) is more likely than not to be non negattgaeive. Since the logarithm is strictly monotonic, the following inequality tends to hold (more accurately, holds in expectation; see I5 empirical evidence):
To(y) Wshort) no (y Fiong), for y mo (y(
(62)
34

---

Published as a conference paper at ICLR 2026
b. Applying SoLo-RA to yt may harm long-context capability. loss becomes:
Referring to Table
| the SoLo-RA
B l
Tsshor)) Ihe (y/ (Ttony)
SoLo-RAsimPO =E a. By y (uloor)
log
(63)
( Wshort)
Tref (ye (iong)
Sol.o-RADro =F a.. Dy yowo(ullahoree
[8] log
log
log
(64)
To (Tlong)
Tree Tshort)
Axioon)
Th Tahort)
(2long
SoL.-RAoRRO- =E a. my y=
log
+ log
(65)
(1) (Toogg)
Te T (short)
Since the latter two terms SoLo-RApro pro are independent of the learnable parameters 8, and the reward coefficient does not affect the subsequent analysis, we treat the SoLo-RA in DPO as equivalent to that in SimPO. Expanding the expectation, we separate cases based on the likelihood ratio:
(T (Lshar))
Te (vl (ahoort)
SoLo-RADPO/SS PO = z. Dy youra (alaa)))))
1] log
(66)
To (ltong
To (1/ [long)
Tt (1) (Eahort)
Tel [ (ahort))
z. D you (ooot))
log
(67)
No (y) [long)
To (ioog)
no [o((u me (v/l (Tloog)
No To (yll 2 1](log
1 no (y) [iong)
Sol.o-RAORPO 2. D yaxol puxa(ylashonk
+ log
ne (va(yl (long)
ne (1) Tahort))
(68)
1 To (soor))))) To To (t)()))))))
To (y) (shor)))) no To
1 To Tiong)
E e. Dy youk (ylator)))
1](oog
+ log
1 - -o [ Tahors)
(69)
We restrict our analysis to the cases in SoL as SoLO-RAORPO can be examined in the same manner. We next consider two cases:
Case 1: Winning Responses (y yw) Based on Eq. (62). we have Ty (y (Ttony)
T (sshor))) Here, SoLo-RA minimizes log Tuly [Fiong]
nolyal Matorr)
which:
Decrease ne (Nwl (nominator) but is counterbalanced by first loss term (short-context preference optimization);
Increases Te (yw (Toogg (denominator), aligning the objective of long-context alignment.
also includes short-context PO (Eq. (56)), this dominates and can prevent Te (xsoot)))))) from decreasing, thereby mitigating the negative impact on the model's short- context performance.
Since SoLoPO
Case 2: Losing Responses (y = y)) Based on Eq. 62 we have me (3n (Ttogg) S To(y) (Ehort).
SoLo-RA again minimizes log (T kaan)))) leading
Rotgl (
Decrease in Te (yI[=shorz) (desirable for reducing poor generation)
Increase in mo ( (Zoong) (undesirable, as it promotes y! in long contexts).
However, there is other loss here that can reduce the prediction probability of To( stong) therefore, using Case 2 would bring certain negative impacts to the model's long-text capabilities.
Based on the above analysis and our goal of improving training efficiency, we adopt the chosen-only SoLo-RA in practical applications of SoLoPO. The experimental results in Section 4.2 and Figure 2a further demonstrate the effectiveness of the chosen-only SoLo-RA
No conffict with the original objective of SoLoPO It is important to note that the above analysis does not conflict with the original optimization objective of SoLoPO (with both SoLo- RA). As long the objective is perfectly optimized-es e.g. the SoLo-RA loss (Eq. (57) reaches zero the chort-context PO (Eq. (56)) will simultancously reduce To To (I (short) and TT Stong). thereby resolving the issue in Case 2 and aligning with the objective of preference optimization.
35

---

Published as a conference paper at ICLR 2026
d. Empirical validation of relationship (y I'short) 2 Tal TT(y) Foong We further empirically
validate the relationship ( @shoot) 2 (T Zlong) for ~ Te( (Eq. 62 Specifically, 100 preference pairs are sampled from Zshont (length 1K) using Bwen-2.5-7B-Instruue corresponding contexts are then extended to lengths from 4K to 32K in 4K increments to form Flog For each length, measure the proportion of instances in which the relationship holds. As shown in Table 15 the relationship is preserved in 100% of the cases across all context lengths tested.
and the
Table 15: Proportion of cases where relationship <y Fiston) z イ (3) Flong) holds for preference pairs (yw > y/) sampled from short (length 1K) evaluated on Flong with varying lengths.
Length Flong
4K
8K
12K
16K
20K
24K
28K
32K
Ratio of (Br (short) 2 (
Zloog)
100%
100%
100%
100%
100%
100%
100%
100%
Ratio of (p( Tahortt Tyl
Flong)
100%
100%
100%
100%
100%
100%
100%
100%
1.11 SUPPORTING ANALYSIS OF SoLoPO's STABILITY SHORT-CONTENS CAPABILITY
Appendix 1.10.2
analyzes the rationale of the chosen-only SoLo-RA. In Case 1, it is noted that during optimization. Te (iee (ahort)) may decrease. However, since SoLoPO also incorporates short- context PO (Eq. (5)), this term dominates and prevents Ty(yulZshord from dropping, thereby alleviating distributional shift short contexts 16 and mitigating its adverse impact on short-context performance.
I.12 APPLICATIONS OF SHORT-TO-LONG PREFERENCE OPTIMIZATION Loss
Table 16: Variants of Convex function f(x) and upper bound function s(x).
Methods
[xx
sxr)
To(2, v)
DPO 54
log(1 1 + +)
|>| + 2 2oog 1 + e²
Glog *o(ylu)
+ 3 Blog ((x)
(vgr()))))
SimPO 48
log(1 + +
|o| + 2 log(1 + e²r)
M log no (B(l))
ORPO 30
log(1 1 + +
xx| + 2logg + e²r)
log 1- no (vlz)
(
IPO 1
x²
2n² + 27
log ... [v]=)
<> mn)
SLiC 85
max((,, -0)
|0||
log मe (7)
e (m 22) 2e 7 cosh [l))
In Section 2.3 we apply SoLoPO to several mainstream preference optimization (PO) algorithms, including DPO, SimPO, and ORPO. More generally, SoLoPO compatible with any PO algorithm for which there exists upper-bound function s(x) of its convergence function fz) such that
f(z ((x+)) + + S s(x).
(70)
Table 16 lists each PO algorithm with its corresponding fx) and s(z). It is worth noting that if the inequality is tight, the performance would be further enhanced theoretically. If s(x) is perfectly tight, iee. f( + 7) + f(- + 7) = s(x), then s(x) should satisfy the following properties:
s(x) is an even function
Vz, s(x) 2 2 f(7) s(0)
We subsequently present the complete theoretical objectives for all considered PO algorithms.
DPO Setting:
T0(Vvo2))) /3 log
To (2:)
SoLoPO -E 2~ z~Da Paorri
Dogar (3 [ log
7D]
Tref (Yw (x)
Tres r (yi (x)
Vw. D. DyiMu
+ Ex DD. iy 3 To [shore))
Tel (y)])))ng)))))))))))))))))))((
3 log
Tref [shore)
Kres (3) (long)
36

---

Published as a conference paper at ICLR 2026
SimPO Setting:
B Neol
3
SotoPO
CimPo
-E
x short
[log (3
log No (vvo (x))
log No (o( - >
yio M11 yil 11
3
+ Fr -D.. MDD,
DD..
log To (xsoor)))))))
log (gogg)
|y1
1y/
ORPO Setting:
To maintain consistency with the vanilla ORPO, we add the conventional causal language modeling negative log-likelihood (NLL) loss.
(y .x)
no (3 (x)
CORPO
SoLoPO
E g~>De
[log (3 [log 1
To (Bew (r)
log
1 xe (M (x)
3- Ex ~ D.. D. log To Fo(yy To (1) (istort)
(y) [long
log
no (tang)
E EEWD spore (No Dw
IPO Setting:
T0(Vvo
To(30112))
IPO
SatapO
E xr Dor Yo Wur Dithu --
[3 flog
Tre (iee( (x)
log
Tyee (211zx)
7112
To To(yl (o(hors)) Tref (3)
log no To ((tong) 12
+ 18 Ex, De. DD [log
Tree (y)
SLiC Setting:
CSLiC
[max(() (0, - log na (raalhol + log To (wnl + 7)
mhord Un Du DiiDoi
E D.. iw,DD,
log Te (z)(z)horz) log No (8( (ioog)
37


Reviews:
[{"review_id": "1mXvEzaQqu", "reviewer": "Reviewer_N1ah", "summary": "This paper introduces SoLoPO (Short-to-Long Preference Optimization), a general framework for efficiently enhancing long-context reasoning in large language models. The key idea is to decouple long-context preference optimization (PO) into two stages: (1) short-context PO, which focuses on optimizing reasoning and alignment on short, information-dense contexts; and (2) short-to-long reward alignment (SoLo-RA), which aligns reward scores between paired short and long contexts that share essential task-relevant information.\nThe method can be plugged into existing PO algorithms such as DPO, SimPO, and ORPO. The authors derive a theoretical upper bound showing that the long-context PO objective can be approximated through this two-part decomposition, and propose a “chosen-only” SoLo-RA variant that further improves efficiency. Experiments on reasoning (LongBenchV1/V2, NIAH-Plus) and instruction-following benchmarks (MMLU-Pro, GPQA, BBH) demonstrate that SoLoPO achieves higher long-context performance, better reward–KL efficiency, and shorter training time without harming short-context capabilities.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.The proposed decoupling framework (short-context PO + reward alignment) is elegant, theoretically grounded, and easy to integrate into existing RLHF/PO pipelines.\n\n2.The theoretical formulation clearly explains how SoLoPO approximates the long-context objective through an upper bound, providing a solid foundation for the method.\n\n3.The “chosen-only” SoLo-RA variant is an insightful practical contribution that reduces instability and significantly cuts training cost while maintaining effectiveness.\n\n4.Extensive experiments cover multiple backbones (Qwen2.5-7B, Llama3.1-8B) and benchmarks, showing consistent gains in both long-context reasoning and efficiency.\n\n5.The method generalizes well across DPO, SimPO, and ORPO, confirming its broad applicability.\n\n6.The paper is well written, conceptually coherent, and supported by detailed ablations and efficiency analyses.", "weaknesses": "1.The paper could provide more qualitative analysis or visualization to show how SoLoPO improves long-context reasoning (e.g., attention heatmaps or retrieved key information patterns).\n\n2.The framework assumes that short contexts can fully preserve essential information; performance may degrade if the summarization or compression is imperfect, which is not deeply discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjhQesmkYF", "reviewer": "Reviewer_mA21", "summary": "This paper addresses the challenge of long-context alignment and aims to improve context utilization in large language models. The authors propose a theoretically grounded framework that leverages short-context preference optimization to enhance performance in long-context settings. The key idea is to decouple long-context preference optimization into two components: short-context preference optimization and short-to-long reward alignment. This framework, termed SoLoPO, can be applied to existing preference optimization methods such as DPO, SimPO, and ORPO. Experimental results across multiple benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "1. **Missing baseline.** Why does Table 4 not report the performance of the LongPO baseline?  \n2. **Clarification on Figure 4.** What exactly do “efficiency” and “runtime” refer to? It is unclear how SoLoPO improves computational efficiency, as it requires handling both $ x_{\\text{short}} $ and $ x_{\\text{long}} $, which requires additional time for dataset creation as well as an additional forward pass for every iteration. Could you clarify where the runtime or efficiency advantage arises?  \n3. **Clarification on Theorem 1 and the function $ s(\\cdot) $.**  \n   1. Theorem 1 introduces $ s(|x|) $. It would be helpful to provide intuition for the role of $ s(\\cdot) $ and its relationship with $ f $, beyond the theoretical requirement for this assumption.  \n   2. Since $ s(\\cdot) $ must satisfy the inequality, multiple valid functions may exist. Would a tighter $ s $ lead to a stronger upper bound, and how would this affect empirical behavior?  \n   3. It would improve readability to include some intuition, after Theorem 1, on how $ s(\\cdot) $ can be derived for a given alignment method. The appendix (I3, I4) provides examples, but referencing them directly in the main text would help readers.  \n4. **On the equivalence with standard PO.** Line 201 states that for $ \\rho = 100\\% $, SoLoPO is equivalent to the original PO. In this case, we have  \n   $$\n   L_{\\text{PO}}(x_{\\text{long}}) \\le \\tfrac{1}{3} L_{\\text{SoLoPO}}(x_{\\text{short}}),\n   $$\n   which is an upper bound. Is there theoretical evidence that this bound becomes an equality? If not, have you empirically tested whether optimizing PO or its SoLoPO upper bound leads to equivalent results when $ x_{\\text{short}} = x_{\\text{long}} $?  \n\n**Minor comments that do not affect rating**\n1. In Table 4 (Open LLM Leaderboard), best and second-best values are not bolded or underlined, unlike other tables. Any reason for this inconsistency?  \n2. While Assumption 1 appears reasonable, it would strengthen the work to provide empirical evidence that it holds in practice.  \n4. What is the base model used in the LongPO reimplementation?  \n5. In Figure 1a, the second box should read “short-context preference optimization.”  \n6. Line 159 defines $ x_{\\text{short}} $ as concatenating $ c_{\\text{rel}} $, but the equation shows $ c_{\\text{irr}} $. This appears to be a typo.  \n7. Section 4.1 would be stronger if it explicitly stated how many scenarios SoLoPO outperforms the corresponding PO baseline, further reinforcing its general applicability.  \n8. In Table 4, the result for $ M_{\\text{short}}^{\\text{SimPO}} $ in the Math column is shown in red, even though it improves over the Instruct model. Please verify the formatting.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. **Novel framework.** Proposes the SoLoPO framework to transfer short-context preference optimization capabilities to long-context alignment.  \n2. **Theoretical foundation.** The method is supported by solid theoretical results that justify the proposed decoupling. \n3. **General applicability.** The framework can be integrated with multiple preference alignment methods, showing consistent improvements across them.  \n4. **Strong long-context performance.** The chosen-only SoLoPO variant consistently outperforms standard PO baselines on long-context benchmarks.", "weaknesses": "1. **Degraded short-context performance.** The method shows reduced performance on the short-context Open LLM Leaderboard. Lines 103 and 377 claim that SoLoPO maintains short-context performance, yet Table 4 indicates otherwise. SoLoPO underperforms the PO baseline in 16 out of 24 datasets.  \n2. **Limited intuition for theoretical results.** Although the theory is sound, the paper should do a better job providing intuition on *why* the decoupling leads to improved long-context performance (see Question 2).  \n3. **Lack of detail in key arguments.** Some claims and statements would be clearer with additional justification or elaboration (see Question 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QgicArLrwJ", "reviewer": "Reviewer_QMaK", "summary": "This paper introduces SoLoPO (Short-to-Long Preference Optimization), a framework designed to improve the long-context capabilities of large language models (LLMs) by decoupling long-context preference optimization into two components:\n1. Short-context PO: Enhances reasoning on compressed, task-relevant contexts.\n2. Short-to-Long Reward Alignment (SoLo-RA): Encourages consistency in reward scores between short and long contexts containing the same task-relevant information.\n\nThe authors provide theoretical justification for this decoupling and empirically validate SoLoPO across multiple preference optimization algorithms (DPO, SimPO, ORPO) and models (Qwen2.5, Llama3.1). Key results show that SoLoPO improves performance on long-context benchmarks (e.g., LongBench, RULER, NIAH-Plus) while maintaining short-context capabilities and significantly boosting training efficiency.", "questions": "1. How does SoLoPO perform on non-QA long-context tasks such as summarization, multi-turn dialogue, or document-level translation?\n2. Can the framework be extended to generation-heavy tasks where both input and output are long? The current focus is on long-input scenarios.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A theoretical decomposition of long-context PO into short-context PO and SoLo-RA.\n2. The idea of decoupling long-context alignment into short-context reasoning and cross-context reward alignment is novel and well-motivated.\n3. Offers a practical and scalable solution with clear efficiency gains (e.g., 2.1× longer trainable sequences, 52% runtime reduction).", "weaknesses": "1. The theory relies on the redundancy hypothesis and Assumption 1, which, while empirically supported, may not hold for all long-context tasks (e.g., when all context is relevant).\n2. The synthetic dataset construction (mixing relevant and irrelevant documents) is simple but may not reflect real-world long-context complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)