LOGO --- Long cOntext aliGnment via efficient preference Optimization
Authors: Zecheng Tang
Zechen Sun
Juntao Li
Qiaoming Zhu
Min Zhang
Venue: ICML 2025 poster (2025)
Keywords: long-context alignment
long-context model
reinforcement learning
Primary area: deep_learning->large_language_models
TL;DR: This paper introduce a novel and efficient long-context alignment method, LOGO, which can be applied to both context window scaling and long-context alignment scenarios.

Abstract: Long-context models (LCMs) have shown great potential in processing long input sequences (even more than 100M tokens) conveniently and effectively. With significant progress, recent research has pointed out that LCMs can accurately locate token-level salient information within the context. Yet, the generation performance of these LCMs is far from satisfactory and might result in misaligned responses, such as hallucinations. To enhance the generation capability of LCMs, existing works have investigated the effects of data size and quality for both pre-training and instruction tuning. Though achieving meaningful improvement, previous methods fall short in either effectiveness or efficiency. In this paper, we introduce LOGO (Long cOntext aliGnment via efficient preference Optimization), a training strategy that first introduces preference optimization for long-context alignment. To overcome the GPU memory-bound issue caused by the long sequence, LOGO employs a reference-free preference optimization strategy and adopts a position synthesis method to construct the training data. By training with only 0.3B data on a single 8 x A800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K model to achieve comparable performance with GPT-4 in real-world long-context tasks while preserving the model's original capabilities on other tasks, e.g., language modeling and MMLU. Moreover, LOGO can extend the model's context window size while enhancing its generation performance.

LOGO -
Long cOntext aliGnment via efficient preference Optimization
Zecheng Tang" 12 Zechen Sun 12 Juntao Li
Qiaoming Zhu 12 Min Zhang 12
Abstract
video understanding (Weng al., 2025). Additionally, long- context models (LCMs) climinate the need for complex toolchains and intricate workflows, e.g., RAG (Yu et al., 2024), that were previously required to address context- length constraints (Ravaut et al., 2024), facilitating tasks such as long-document summarization (Laban et al., 2024) and project code analysis (Zhu al., 2024).
Long-context models (LCMs) have shown great potential in processing long sequences, with re- search showing they can accurately locate token- level salient information. Yet, the generation per- formance of these LCMs is far from satisfactory and might result in misaligned responses, such as hallucinations. To enhance the generation capabil- ity, existing works have investigated the effects of data size and quality for both pre-training and in- struction tuning stages. Though achieving mean- ingful improvement, previous methods fall short in either effectiveness or efficiency. In this paper, we introduce LOGO, an efficient and effective training strategy that first introduces preference optimization for long-context alignment. LOGO consists of reference-free preference optimiza- tion strategy and a corresponding efficient data synthesis process. By training with only O.3B data on single 8x GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instren 80K model to achieve comparable performance with GPT-4 in real-wordd long-context tasks while preserving the model's 's original capabilities on other tasks, e.g., language modeling and MMLU. Besides, LOGO can also scale the models' context window size while enhancing their performance.
Yet, recent studies have pointed out that open-source LCMs failed to achieve satisfactory performance in real-world or complex synthetic long-context tasks, where LCMs might produce misaligned results, such as instruction unfollowing and hallucinations (Tang et al., 2024; Zhang et al., 2024a). To mitigate the above issues, the open-source community has made significant strides, primarily focusing on building and scaling up high-quality synthetic data post-tune the models (Wu et al., 2024a; Bai et al., 2024; Fu al., 2024; Li et al., 2024). These efforts, shown in figure 1, have led to notable improvements but still fall short in terms of either effectiveness or efficiency. For instance, the Alama-3.1-88. 128K model (AI@ Meta, 2024a), trained on over 800B long- instruction data, still underperforms the Llama-3-8B-800 model (Zhang al., 2024b), which was post-trained with only 1.5B high-quality data. However, the Llama-3-8B-800 model shows only slight improvement over the backbone and still lags far behind closed-source models like GPT-4.
To investigate the reasons behind LCMs generating mis- aligned outputs, we visualized the information retrieval ca- pability (reflected by Retrieval Score) and the generation capability (reflected by Recall Score) of different LCMs synthetic retrieval task'. As shown in figure 1(b), we can observe minimal difference among the retrieval scores of various LCMs, but a large difference in their generation performance. This suggests that while LCMs are adept at identifying key information within long contexts, they struggle to effectively utilize the retrieval information for generation. The underlying cause might stem from the fact that these methods primarily focus on optimizing the train- ing data for the Supervised Fine- Fine-Tuning Tuning (SFT) stage, which aims to enhance the adherence of LCMs to long instructions
1. Introduction
Processing long input sequences is a fundamental capabil- ity for Large Language Models (LLMs) nowadays, with a few models being able process context lengths exceed- ing millions of tokens (Team et al., 2024; MiniMax al.,, 2025). This capability unlocks the potential of LLMs for novel tasks and cutting-edge applications, such as high- resolution image processing (Tian et al., 2024) and long
'School of Computer Science and Technology, Soochow Uni- versity Key Laboratory of Data Intelligence and Advanced Com- puting, Soochow University. Correspondence to: Juntao Li <ljt@ssuda.cao suda.cdu.cn...
Retrieval capability is reflected through the recall score of salient tokens located by retrieval heads (Wu et al., 2024b). We calculate the average recall score across the top-10 retrieval heads. A higher retrieval score indicates that the LCM can retrieve more critical information. Details are shown in Appendix A.
Proceedings of the 4 International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LOGO Long cOntext aliGnment via efficient preference Optimization
GPT-4-Turbo
57.9
60
Retrieval Score
50
Recall Score
47.9
Claude-2.1
55.8
42.3
40
39.2
GPT-4
48.5
30.4
30
(Ours) Llama-3-8S-LOGO
47.0
20
Llama-3-8B 3-8B-Instruut-BOK
42.3
10
GPT-3.5-Tlurbo166
42.0
10
Liama Instruct
40.7
20
11.7
12.0
122
12.7
LongChat- V1.5-32K Lamma-311-8-122
Llama-3-8---8 Mama-3-88-LOO8
39.2
(b) Retrieval score and Recall score of LCMs
(Ours) Llama-2-7B-L0G0
35.1
50
Mistral instrcct 7B V0.2
31.1
After post-training Backbone model
0.38
45
1.5B
(ours)
LongChat-V1.5-777
30.4
Llama-2-78-80K
29.0
35
2B
Llama
27.6
30
L'ama-2-based Model
Long- context Foundation Model
25
Ulama-3-based Model
Model
LongChat-V1.5-322 Llama-3.1-88-128. Llama-3-88-B8 Llama-3-88- 1ama----B-LLGA
(a) Performance on real-world long-context tasks
(c) Model Performance V.S. Long-context Training Data Size
Figure 1. (a) Performance of widely-used LCMs real-world long-context tasks, where LCMs feature large context window size but relatively poor performance; (b) Retrieval score (information retrieval ability) and Recall score (generation ability) of LCMs on the synthetic retrieval long-context task (multi-value NIAH); (c) Long-contex (pre-) ppre-)training data size for each LCM.
but lack further preference-aligmman training to further calibrate the model's 's outputs. Additionally, given that the sequence length of the context is typically much longer than the prediction portion, the feedback signal during the SFT stage, i.e., cross-entropy loss, from the prediction is often overshadowed by that from the context. Thus, merely optimizing LCMs with inadequate SFT data is insufficient.
GPT3.5-Turbo in real-world tasks and approaches the per- formance of some top closed-source models like GPT-4. Additionally, LOGO can also benefit the context window scaling stage of short-context LLMs such Llama-2-7B. Chat-4K (Touvron et al., 2023), where we can extend their context window size up to 8 times (e.g.,32K context window size for Llama-2-7B-Chat-4K and simultaneously enhanc- ing their performance substantially.
In this paper, we propose LOGO, an efficient and effective Long cOntext aliGnment training strategy based on prefer- ence Optimization that is suitable for both context window scaling and long-context alignment. Two key components are involved in LOGO: (1) an efficient preference data syn- thesis pipeline tailored for Long-contedt scenarios, and (2) a long-context preference optimization training objective. is worth noting that training with long sequences is a cemmrry-ineessine task (Dao, 2023) and the DPO algorithm also has a high GPU memory demand. Thereby, LOGO adopts reference-free training objective and the positional indices synthesis method (Zhu et al., 2023) to overcome the GPU memory- bound and improve the training efficiency.
2. Related Work
2.1. Context Window Scaling and Long-context Alignment
Two steps are essential for empowering LLMs with the abil- ity to handle long-context tasks: I) context scaling, which expands the limited context window size to support long- context tasks, c.g., from 8k to 128k; and 2) long-context alignment, which ensures that LCMs can follow long in- structions. Currently, the open-source community mainly focuses the former, primarily by (1) post-training models long instruction data (Chen et al., 2023b; Xiong et al., 2023; Fu et al., 2024; Zhang et al., 2024b), (2) devising novel model architectures (Yang et al., 2023; Zhang, 2024; Tworkowski et al., 2024), and (3) modifying positional en- coding (Peng et al., 2023a; Chen et al., 2023a; Jin et al., 2024) to extend the context window of LLMs. However, current works (Tang et al., 2024; Hsieh al., 2024; Zhang
By training with LOGO, LCMs can achieve significant im- provements in real-world tasks and gain moderate improve- ments in synthetic and language modeling tasks, as well as maintaining good performance on the short-context tasks, e.g., MMLU (Hendrycks et al., 2020). As shown in fig- ure ((a), our Llama-3-8B-LOGG significantly outperforms
2

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
al., 2024a; You al., 2024) indicated that LCMs still underperform in long-context tasks, frequently manifesting issues such as hallucinations and failure to follow instruc- tions, despite possessing large context window size. To mit- igate this issue, Bai et al. (2024); Wu et al. (2024a); Chen et al. (2024) proposed to align the LCMs in long-context scenarios by synthesizing long-dependency instruction data to fine-tune the models. Some LLMs are even pre-trained with massive long instruction data (Jiang al., 2023; Dubey et al., 2024; Abdin et al., 2024). Yet, despite numerous attempts that have been made to improve the data quality and quantity, the performance of open-source LCMs still lies far behind close-source LCMs. In this work, we tackle the above challenge by rethinking the long-context training objective. We introduce an efficient long-context preference optimization training strategy, i.e., LOGO. With a small amount of data and computational resources, LOGO can significantly enhance the model performance of LCMs.
proved variant of DPO, which employs an implicit reward formulation that directly aligns with the generation met- ric, e.g., PPL, thereby eliminating the need for a reference model. The training objective of SimPO can be written as:
B (Mw
Lsim = - -Z( un)
log
log To (Vwlz)
3 1yal
log To (z) (
where To is the policy model, 3 (scaling of the reward differ- ence) and y (target reward margin) are the prpeepparameters to separate the preferred and dis-preferred responses.
Positional Indices Synthesis Transformer-baat based models rely on positional indices to identify the relative position of each token (Raffel et al., 2020). One efficient method to extending the model's input context length is to adjust the positional indices, simulating long-sequence inputs with- out modifying the actual input sequence (Press et al., 2021; Ruoss et al., 2023). By default, the positional indices of a sequence of length K are P(K) = {0, 1, 1.... ,K- - 1}. To extend the sequence length from K to K", we can syn- thesize the positional indices: P=(K')
2.2. Direct Preference Optimization
Direct Preference Optimization (DPO) (Rafailov et al., 2024) is widely adopted RLHF algorithm (Ouyang et al., 2022) that aims to align models with human preferences. Compared to other reinforcement learning methods, e.g., PPO (Schulman et al., 2017), DPO can achieve strong per- formance while eliminating the need for a separate reward model. Different from Supervised Fine-Tuning (SFT) which models align with ground truth at the token level, DPO up- dates model parameters based on discrete evaluation scores. Specifically, DPO guides the model to "reject" misaligned responses and "accept" preferred responses with differently assigned prediction scores. Significant efforts have been made to improve the DPO, such as RSO (Liu et al., 2023), CPO (Xu et al., 2024), TPO (Saeidi al., 2024), and ORPO (Hong al., 2024). Compared to short-context tasks, obtaining DPO data for long-conten tasks is consid- erably more challenging due to the absence of open-source evaluation models designed for long-context tasks and the complexities involved in manual annotation. Thereby, we propose an efficient and theoretically guaranteed method for long-context DPO data synthesis.
{0 bo. 1 + b,,-.. K - 1 + bk-1). 1.. where B indicates the positional bias {bu, d1. index and K 1 + bx- = K". To ensure effectiveness, the synthesis of position indices should achieve a uniform dis- tribution of relative distances within the extended sequence length [0, K"] and cover as many of the extended positional indices as possible (Wu et al., 2024a).
.x-1) applied to each original position
3.2. Long-context alignment with LOGO
3.2.1. TRAINING DATA SYNTHESIS OF LOGO
Given the challenges of preference data synthesis for long-context tasks, we begin by presenting the process of LOGO's s preference data synthesis. For each long- context sample, we can format it a triplet (Q,C, P). where Q, C. and P represent the question, the reference long-context, and the model prediction, respectively. As shown in Fig. 2, we first divide C into equal-length chunks {CI, : C.). Then, three steps are involved: (1) Im- portance Scoring with Automatic Evaluator, (2) Preference Data Synthesis, and (3) Positional Indices Synthesis.
3. Methodology
3.1. Background
Direct Preference Optimization Given prompt x, DPO aims to maximize the likelihood of a chosen response yo over a rejected one y/, thereby preventing the model from generating undesired content. There are three essential mod- ules in DPO: one frozen reference model and one trainable policy model for calculating the DPO loss jointly, and one evaluation strategy (or evaluation model) for distinguishing between you and yı- SimPO (Meng et al., 2024) is an im-
Importance Scoring with Automatic Evaluator To con- struet preference data in the long-context scenario, an ef- ficient method is to guide the model to respond based on contexts with varying degrees of influence on the outcome. More concretely, to construct the chosen data, we only pro- vide the model with context relevant to the question, thus enhancing the fidelity of the model's output by reducing contextual interference (Shi et al., 2023). Conversely, we
3

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
! Importance Scoring with Automatic Evaluator
Q
Ques tion Prediction
C
Chunk
Preference Prediction
Rejected Prediction
Full Length Contect Instruction Data
score:
sccure: Entity Overlap bettween C, and Q
iscure:
Iscore: 0
(val())
Positional Indices Synthesis
Preference/DDis-Preeeenn Data Synthesis
Skapped
Skipped
g Preference Prediction (PP): Inference w/ uritical C
Chunk
Added Iereverent chunk
Chanix
of Top-N highest scores
Cy)
XM
Dispprfeeence Sample
score: 1
score: 0
Slapped Chunk
Skipped
G
CN
Added Inreverent chunk
Chunk
of Top-N lowest scpres
Dis-preference Prediction (DP): Inference w/o critical
Cy
x1
score: 6
score: Cy
Preference Sample
T Dis- prefereeeeee Prediction (DP): Inference w/ random C
>
C. of N random scores
Synthetic Positional Indices
Figure 2. Training data synthesis process of LOGO, where we guide the model to generate preference data based on the detected critical segments within the long context and apply positional synthesis strategy to "extend the sequence length".
utilize irrelevant context or reduce key information to guide the model in generating misaligned content like hallucina- tions. To find the relevant chunks C, within the context, we utilize an automatic evaluator Eval(.) to measure the "contribution" " of each chunk C₁ to the question Q. Specif- ically, we identify all the entities within a chunk C, and calculate the overlapping entities between each C, and Q, where a greater overlap indicates higher influence of C, to Q. Thereby, we can efficiently assign importance scores sS {s1, $2 Sn to all the chunks.
relevant chunks (C=sUC>a)) Subsequently, the constructed Pehosen and P Projected share the same context C', which is constructed from all the chunks in C>. and partial chunks in Ces. Finally, one LOGO training sample can be written as [[Q,C C. ), (Q, C' p(i) 1. li-1), where M is the rejected sample number in equation 4.
rejected
Positional Indices Synthesis Given that each preference data includes (M + 1) instances, with one chosen sample and M rejected samples, a long context length of |C'| can easily lead to GPU out of memory (even with 80GB mem- ory). To address this, we employ a positional encoding synthesis strategy. By assigning different synthetic posi- tional indices to each chunk, we can simulate long-sequence imput with just short context (Wu et al., 2024a). Specifically, to ensure that the synthetic positional indices do not disrupt the semantic structure of short context, the positional indices within each chunk should be continuous, while indices be- tween adjacent chunks can be discrete, i.c., omitting certain positional indices (as shown in sub-Fig. 3 in Fig. 2). Given N equal-length chunks within each sample", to achieve uniform distribution of relative distance within the ex- panded context length [0, K'), each positional bias term b, B should be sampled from uniform distribution. The synthetic positional indices can be written as:
Preference Data Synthesis Let N represent the number of chunks within context, we define a threshold 8 to filter the critical chunks. Specifically, chunks C>s scoring above ō are considered as critical chunks while chunks Cca scor- ing below 8 are considered irreverent chunks. Then, we combine Q and 8 for model to generate correct prediction Pehosen. and adjust the ratio of chunks sampled from Cy.8 and Ces for model to generate wrong predictions Prejected-
is worth noting that the open-source community currently lacks effective methods effectively identify models' mis- aligned outputs for the long-context tasks, which poses great challenge for selecting chosen and rejected samples in preference data synthesis². Therefore, instead of construct- ing one rejected sample with a specific error pattern, we can expand the rejected space to push the model away from a range of possible wrong predictions. Specif- ically, we construct Prejected based on two misaligned er- patterns: (1) model generation based on all irrelevant chunks (C=s), and (2) model generation based on partially
(P&(K) = (i bi where b, ~ U(1, (i mod (C,) x (K' K)/N),
(2)
"Since the length of question Q and prediction P are much sharter compared to the long context C, can ignore the length of Q and P for simplicity.
We analyze the method of utilizing GPT-4 evaluation model and show some misalignment cases Appendix B.
4

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
where (i mod (C)) indicates the chunk index where the cur- rent positional index i resides, and (K' K)/N represents the expansion size for each chunk.
and Summarization; (2) 2,000 instances sampled from Red- Pajama (Computer, 2023) mitigate forgetting, where we prompt the open-source LCM Qwen2-70B-Instruce (Yang et al., 2024) to generate questions for each instance. Then, we split each instance into equal-length chunks, with each chunk containing 512 tokens. To construct preference and dis- dis-preference data, we use the spaCy model", a named entity recognition (NER) model that can identify all the entities within context, the evaluator Eval(.). We use the number of overlapping entities between each chunk C; and the question Q as the importance score. We set the threshold 8 as 6, chunk number N as 16, and | = 512. For positional indices synthesis, we adopt two different sampling strategies on positional bias to ensure that all positional indices are uniformly covered and maintain the semantic structure of the context. Finally, we have a total number of 12,000 training samples, with a total data size of approximately 1 12,000xx122 16x3 tokens.
We provide theoretical guarantee that the synthetic po- sitions can cover all possible scenarios given an adequate amount of data. Further details on the positional synthesis process are provided in Appendix E.
3.2.2. TRAINING OBJECTIVE OF LOGO
The training objective of LOGO can be written as:
(LoGO(TT) [(Te) + (z..). log (z))lzz))
where ((TT)) is the preference optimization term, À is the hyper-parameter, and (,,,,.)) log Na (aw))r)) is the SFT reg- ularization term*.
As mentioned above, we expanded the rejected sample space to address the issue of lacking long-context evaluation mod- els. Consequently, we guide the model to simultancously reject multiple potential rejected samples to construct the preference loss objective. Thus, [(TT) can be written as:
Training Details To validate the effectiveness of our method while controlling experimental costs, we use the efficient fine-tuning method LoRA (Hu et al., 2021), which fine-tunes only the attention and token embedding modules. We set M as 2 in equation 4. Due to the positional indices synthesis, LOGO can potentially scale the context window size and ensure alignment in long-conten, tasks simulta- neously. To demonstrate the effectiveness of LOGO, we experiment with two types of models: (I) Short-context Models (SCMs) including Llama-2-7B 2-7-a-2-7B-hha -hCat (Touvron et al., 2023) and Llama-3-8B-Instrene (AI @ Meta, 2024b), which own context lengths of 4K and 8K, respectively; and (2) Long-context Models (LCMss, including Llama3-8B- Instruct-80K (Zhang et al., 2024b), Llama-2-74 2-7B-Instrucct 80K (Fu et al., 2024) and Mistral-Instruct-71 et al., 2023), which inherently have long context windows. For SCMs, we scale the context windows to 8 times their original context length. We set X = 0.1 in Eq. 3 and search the hyper-parameters of equation 4 based on (Meng et al., 2024) for different models, where 3 = 10, 7 = 3 for Llama- 3-8B-based model, 3 2.5, 7 = 0.25 for -istral-Instruct- 7B-V0.2-based model, and 8 = 3, 7 = 0.6 for Llama-2-7B- based model. More training details are in Appendix F.
[(TO)= -B ( (a. of (n) log o ( |Narl
B
log To(yml (z)
(4)
M log no (v) (1)x)) - )
B
M|3
where M is the number of rejected samples, and the remain- ing terms are consistent with those defined in equation 1. It is worth noting that [(TO) is also free of the reference model and more aligned with the generation tasks which is efficient for long-context training. We have theoreti- cally proven that ((TO) has a small generalization error and shows a reliable model performance on the unseen data in Appendix C and conduct the convergence property analysis from the gradient perspective in Appendix D.
(Jiang
4. Experiment
4.1. Settings
Training Dataset Construction We construct the syn- thetic datasets based on two corpora: (1) 4,000 instances sampled from long-Ilm-data (Zhang et al., 2024b), which includes reference contexts from multiple domains (e.g., biography, paper, etc.) and questions generated by GPT-4 covering tasks such as Single-
Evaluation and Task Selection We assess the LOGO training strategy across real-world long-context tasks and the synthetic retrieval task. Additionally, to explore the impact of LOGO training for short-context tasks, we evalu- ate models on MMLU (Hendrycks et al., 2020) and Truth- fulQA (Lin et al., 2021) tasks. For a comprehensive com- parison, we evaluate LOGO against existing widely-used methods from two perspectives: (1) direct context window scaling, including YaRN (Peng et al., 2023a), PoSE (Zhu
Single-Detail Detail QA, Multi-Detail QA,
"The regularization term serves two key purposes: (1) to pre- vent reward hacking (Yuan et al., 2024; Hong et al., 2024), and (2) to ensure that the policy model To maintains its original capabili- ties acquired through SFT/pre-trainin without drifting away. Since ((TT)) is similar to the SimPO loss function.
pittpa://spasee i i ////uuage/mmee
5

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Table Evaluation results LongRench benchmark. We comprchensively compare I.OGO with different strategies, including Training- Free (Free), SFT and DPO. The comparison is also conducted under two settings: (1) context window scaling on SCMs and (2)
long-context alignment LCMs. LOGO achieves the best performance under all the settings.
Models
Type
S-Doc QA
M-Doc QA
Summ
Few-shot
Synthetic
Avg.
GPT-3.5-
39.8
38.7
26.5
67.1
37.8
42.0
GPT-
45.1
55.0
28.3
72.3
41.8
48.5
Loonghha-v1.5-77--5
28.7
20.6
26.7
60.0
15.8
30.4
1.ama-3.1-9B-Instuuct
23.9
15.8
28.9
69.8
57.5
39.2
Results on SCMs (scaling x context window)
Llama-3-8B Instruct-SK
39.3
36.2
24.8
63.5
39.9
40.7
YaRN-64K (Peng et al., 2023b)
Free
38.0
27.4
40.9
40.9
PoSE-64K (Zhu et al., 2023)
SFT
34.9
31.4
18.7
59.3
44.2
37.7
+ LOG0-64K
DPO
39.8
36.7
28.8
65.4
49.0
43.9
Llama-2-78 1anna----BBChh----- Chatt-KK
24.9
22.6
24.7
60.0
5.9
27.6
Data-Enn Data-Engincering Engincering-8OK (Fu et al., 2024)
SFT
26.9
23.8
21.3
65.0
7.9
29.0
LOGO-32K
DPO
26.7
23.3
26.3
63.1
11.1
30.1
Results on LCMs (preserving original context window)
Llama-3-8B-Insiie InSrructttKK LongLoRA (Chen al., 2023b)
43.0
39.8
22.2
64.3
46.3
42.3
SFT
39.3
36.2
26.8
63.5
48.0
42.8
SimPO (Meng et al., 2024)
DPO
40.7
23.5
48.4
44.5
+ LOGO-80K
DPO
44.0
41.2
28.1
68.6
53.0
47.0
Llama-2-7B 2-7B-44K
28.3
33.2
13.4
62.3
28.7
LongAlign (Bai et at,, 2024)
SPT
32.7
26.5
16.5
33.9
-.000-64K
DPO
33.6
28.0
29.4
65.1
24.5
36.1
Sestral-Instruut7B---Y V0- FILM-32K (An et al., 2024)
31.7
30.6
16.7
58.4
17.9
31.1
SFT
37.9
34,9
25.3
64.7
31.2
38.8
LOGO-32K
DPO
38.3
37.6
26.1
67.0
31.5
40.1
al., 2023) and lata-enninn (Fu et al., 2024); and (2) long-context alignment, including LongLoRA (Chen al., 2023b), LongAlign (Bai et al., 2024), and FILM (An al., 2024). Due to space limitation, we report more experi- mental results, including model performance on language modeling task, results of scaling to longer context window size, and more short-context tasks in Appendix G.
context tasks. For instance, PoSE causes the Llama3-88 Instruct model to drop around 3 points average compared to the bascline, with particularly notable declines in perfor- mance on the summarization tasks. For LCMs, LOGO can significantly improve model performance, with all LCMs showing varying degrees of improvement, e.g., Llama-3- 8B-Instruct-S00 model shows an average 5-point improve- ment compared to the baseline, whereas the instruct tuning method tends restrict even well-performing L.MMs to limited performance bottleneck; (2) Compared to other methods, LOGO demonstrates significant improvement in information-intensit tasks, which require the model to gather information from various parts of the context. Specif- ically, summarization and synthetic tasks, LCMs trained with LOGO achieve an improvement at least points.
4.2. Performance on Long-context Tasks
Results on Real-world Long-context Tasks We evaluate the LOGO performance on LongBench (Bai et al., 2023), comprehensive benchmark suite encompassing 16 distinct datasets spread across task categories, including Single Document QA (S-Doc QA), QA), Summarization (Summ), Few-shot, Synthetic, and Code. It is worth noting that we exclude the Code category since the code testing data primarily involves contexts of just around 4,000 tokens and our training data does not cover this domain. As shown in table 1, where we can observe that: (1)
QA (M-Doc
Evaluation Results on Synthetic Retrieval Task To in- vestigate whether the LOGO training strategy affects the information retrieval capabilities of LCMs, we conduct Needle-in-a- Hay--Hysttack (NIAH) testing (gkamradt, 2023). As shown in figure 3, we can find that LOGO can scale the con- text window for SCMs (left group) and does not ad adversely af- fect the original context window size of LCMs (right group). We can also observe that the original LCMs (middle group) and those trained with LOGO (right group) share similar
LOGO achieves the best performance among all the sub- tasks Specifically, for SCMs, LOGO outperforms both training-free and fine-tuning methods. Although these meth- ods can potentially extend the context window of SCMs, they significantly impair performance on real-world long-
6

---

LOGO- Long cOntext aliGmment via efficient preference Optimization
Llama-3-88 -------sttttt-8K > 64K (LOGO)
Llama Malma-3-8B-iae --B-Instruct-00
Llama-3 3lama-3-8B-Instrucee -8-Insttruct8OO0 (LOGO)
0%
0%
0%
22%
22%
22%
44%
44%
44%%
168%
Orignal
Scaled Content
Griginal Contect Lengh
Contoxt
88%
8K
16K 24K 32K 40k 48K 56K 64K
72X BOK
6K
16k 24k
32K
40X
48K
56k
64K
72K
B0K
8K
168
24K
32K
40K
48K
S6K
64K
72K
90X
Llama-2-7B-Chal 32K (LOGO)
Llama-2-7B 1--7--nsttruct8000
1aama-2-7BIInstrut--000 (LOGO)
0%
0%
0%
22%
122%
22%
44%
44%
40%
66%
666%
666%
Griginal
Originl
188%
Conteat
188%
Contect Length
Length
10K 24K 32X 40K 10K 50% 64K
72K
00k
0K 10K 21K 32K 10X 48K 50k 6AK
72K 00X
16K
24X
32K 40K 46K
56k
64K 22k B0K
Context scaling with LOGO
Context scaling with long instruction tuning
Long- Long-context alignment with LOGO
Figure Results of NIAH testing, where green and indicate successful and failure recall, respectively. The test covers context lengths from 8K to 88K with incremental intervals of 0.5K. The needle is positioned at varying depths.
patterns, i.e., similar shades of color, yet LOGO improves performance in areas where the original LCMs fail. This indicates that LOGO does not compromise the inherent capa- bilities of LCMs but rather enhances their original weakness. We can also find that -lamm-33-B--8- model demonstrates superior context scaling results compared to Llama-2-7B 4K. This can be attributed the larger RoPE base value of Llama-3-8B-8K (500,000) compared that of Llama-2- 7B-4K (10,000), which has been shown to facilitate context window scaling effectiveness (Su et al., 2023).
Long-conteat Scaling (x8)
Baseling LOGO
Long-corneet Aligament
Instruct Tuning
LOGU
MELLU
TruthfulQA
05
60
"elignment
60
LOGO
55
50
40
Llans: at
Inttrstt (e8)
35
Llans: 2.78-
Glar (4k)
Insract (a05)
Lians: 2.7
Incerace (B9k)
Misral
vitz (3sk)
Laana sop
Insinct (9K)
Liane 2-7p.
Chat (41)
Unna- mt
Instruct (801)
Lianz 2.7p
Incenet (Rek)
Ristral-7
voz (BOR)
Llaro
4.3. Performance Short-context Tasks
MMLU (Hendrycks et al., 2020) and TruthfulQA (Lin et al., 2021) are two representative short-context tasks that assess LLMs' foundational capabilities. As shown in Fig. 4, we find that LOGO not only preserves the LLM's inherent capabilities on short-context tasks but also demonstrates improvements. This because LOGO aims to teach the model generate responses based on the context rather than fabricating results (such producing hallucinations), which equally applicable to short-context tasks. We also find that scaling context length with LOGO yields better results than instruction tuning. For instance, in TruthfulQA task, Llama-3-8B- ---t-----nttutt-BII shows significant performance degradation compared to our Llama-3-8B-LOG phenomenon indicates a high "alignment tax" paid from instruction tuning (Fu al., 2023).
Figure 4. Model performance on short-context lasks, where the LOGO training objective can preserve the backbone model capa- bility while long-context SFT might impair the model performance on short-context tasks.
iment, as it exhibits strong baseline performance across wide range of tasks. The ablation study focuses on real- world tasks, reporting the average score on LongBench (LB), and evaluates language modeling performance by calculat- ing the perplexity (PPL) score on the PG19 test (Rae et al., 2019a) with context length of 64K. Due to space limitation, we put the detailed analysis between the LOGO and the SFT training strategies in Appendix H and highlight the efficiency superiority of LOGO training in Appendix
Such a
5.1. Analysis of LOGO Training Objective
5. Ablation Study
Effect of SFT Regularization Term À To investigate the impact of SFT regularization term in equation 3, we adjust the value of À from 0.0 to 1.0. As shown figure 5(a), we present a scatter plot visualizing the model's 's performance under different À settings. We observe that increasing
We analyze the impact of various papper-prameterss in the LOGO training objective Sec. and discuss the influ- ence of positional indices synthesis in Sec. 5.2. We mainly utilize the Llama-3-8B 1-S8--Instruct8800 model for the exper-

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Modeling Task
Language
14-2, 2=0.0, Ct [LB: 46.3, PPL: 12.0]
Better Performance
90000
A800 Memory Limit (81251MB)
world Tosts
90000
0.30
(Lower
better)
M=3, 1=00
70000
[LB 47.9. PPI: 11.9)
0.25
60000
M:3, 2:0.1 CbeedK [LB 46.1, PPL: 11.50)
50000
M=2 2-0.1. doced
40000
LLB: 441,, PPI: 11.20]
M-3, A- Cou GPU DOM
30000
0.15
M-1, 2-0.2 RK
M-2 2. -B LB: 47.0, PPL: 10.05)
20000
44.3, PPI: 10.11
0.10
10000
Mr2, 2m0.5, GlamBK [LB: 46.9 11.09)
M=2. Jabli COL-8K
Mal, 1=0.1, Cix-
Ma2. Ma2..e0..1 Ao0.1, Gboodff
M=2, M=2,A00. As0.J, Cix=8K 8K
Mal. 1-0.5, CDx-BK
MeZ, 1 coc-tik Ma3, 2-0.0, Coc-lik Mc3, As10.1, CteedK
M-2, 2=1.0 Cle-8K [LB: 45.7, PPL: 7.44]
0.00
BRetzrr perjormance on Longuage Modielin Trokk
-2
2
4
8
10
Real World Tasks (Larger I.B is better)
Reward dift. (xx,,) Ri 7(
(a) Language Modeling and Real-world Tasks
(b) Reward diff. distribution
(c) GPU Memory Consumption
Figure 5. (a) Comparison among different settings on the language modeling task (PPL. and real-world tasks (Avg. score on LB)); (b) Reward difference distribution under different M settings; (c) Training GPU memory consumption of different settings.
allows the model to achieve a lower PPL score, while the change of À value have minimal impact on performance for real-world tasks. For example, for the three sets of experimental results at (M = 2, A = 0.1, Ctz. = 8K). (M- = 2, X- 0.5, Ctx. = 8K), and (M- 2, = 1.0, Ctr. = 8K), we can observe that as gradually in- creases, the PPI. score decreases significantly by nearly 3.5 points, whereas the average score on LongBench differs by only around 1.5 points. This indicates that LOGO is robust to the SFT regularization term, and incorporating it can enhance the model's language modeling capability.
is crucial. We extend the LOGO with more rejected sam- ples by applying the context parallelism strategy, i.e., ring- attention (Liu et al., 2024), and provide a detailed analysis of this computation-
computation-performance performance trade-off in Appendix I.
5.2. Analysis of Positional Indices Synthesis
Positional Indices Synthesis aims to simulate longer se- quence lengths by synthesizing positional indices while keeping the original input tokens unchanged. To assess the impact of different synthetic lengths on training perfor- mance, we consider two settings for synthetic data length: extending from real input length of 4K to a target length of 64K (Ct.z. = 4K) and from a real input length of 8K to a target length of 64K (Clx. = 8 8K). We keep the chunk size [C,] constant and set the number of chunks as and 16 for the two respective settings. As shown in figure 5(a), short- context synthetic data length significantly diminishes the model's performance on both the language modeling task and real-world tasks (data point (M = 2, A 0.1, Ctz. = 4K) versus data point (M = 2, À - 0.1, Ctr. = 8K)), but can still overcome the instruction tuning method (42.8 aver- age score on LongBench) and effectively reduces the GPU memory requirement during training (figure 5(c)). This is because when the original context length is relatively small (4K), it requires scaling up by a larger factor (16 times) to reach the desired context length (64K). With the same amount of data, some positional indices may be missed or infrequently activated, thereby affecting performance.
Impact of the Rejected Samples Number We experi- ment with different rejected sample numbers M = 21) in equation 3. Specifically, when M equals 1, the LOGO Objective degenerates into the SimPO objective. As shown in figure 5(a). for real-world tasks, more rejected sam- ples can enhance the model capability; while for the lan- guage modeling task, it slightly affects the model per- formance. We also visualize the learned reward margin M r(a, vmm) M7 Eiz D vi ()) under various M values in figure 5(b). We can observe that using a larger M can flatten the distribution and make it easier for the model to distinguish between preference and rejected samples as the gap between r(z, yi) and 1 ΣM r(z, ( gradually in- M creases with larger M. This is because increasing M can cover more samples with various misalignment patterns. However, as illustrated in figure 5(c), increasing the value of M presents a challenge, as it may exceed GPU mem- ory constraints. is worth noting that striking a balance between computational efficiency and performance gains
8

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
6. Conclusion
sorship Program by CAST (2023QNRC001,, and the Pri- ority Academic Program Development of Jiangsu Higher Education Institutions.
In this paper, we find that commonly used training ap- proaches for LCMs, i.e. SFT with a limited amount of data, may degrade the model's generation capabilities, lead- ing to misaligned outputs. To mitigate this issue, pro- pose LOGO, novel and efficient preference optimization strategy for both context window scaling and contecontext alignment. We tackle the challenge of data scarcity in long- context preference optimization by introducing a Positional Indices Synthesis strategy for long-context preference data synthesis. Additionally, we design an efficient and effective preference optimization objective tailored for long- oong-context alignment. With single 8 x A800 GPU machine and just 16 hours of training. LCMs can achieve significant improve- ment in long-context tasks while preserving their inherent capabilities with the LOGO training strategy. Theoretical analysis and comprehensive experiments across various set- tings have validated the effectiveness of our method.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.1422 2024.
@ @ Meta. Llama 3-1 model card. 2024a.
Blob,
URL https: //at meta.. com/ /blgg/ meta- 1-t---1aaaa------- 3-1//
Aetaeta. Llama 3 model card. Blob, 2024b. URL https://githee wow.://ammmommaammmmaaa / /1lama3/ blob/mainn/MCDELLC...
mnd
Aminabadi. R. Y., Rajbhandari, S., Awan, A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Decpspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing. Networking, Storage and Analysis, pp. 1-15. IEEE, 2022.
Impact Statement
This paper introduces a novel Long-cOntext aliGnment strategy via preference Optimization, LOGO. Currently, the long-context model community primarily relies on us- ing larger amounts of pre-traine tee-traiinng data during the context window scaling phase and fine-tuning models with existing mixed long-text instruction data for long-context alignment. However, the benefits of the above approach remain limited. The field still lacks open- spen-source and effective methods for long-context alignment, particularly in the areas of long- preference data synthesis and the development of efficient alignment training objectives. In this paper, during the preference data construction phase, there still remains sig- nificant challenge due to the absence of suitable evaluation models to assess whether the outputs of LCMs are accurate or contain hallucinations. While utilizing higher-quality datasets, such as those created through manual annotation, could improve outcomes, we acknowledge this as an area for further improvement. As an academic paper, we have demonstrated the generalizability of our method through comprehensive experiments and theoretical proofs. We hope that this work will provide valuable insights to the open- source long-context model community, raise awareness of the importance of long-context alignment, and inspire future research with broader societal and technical impacts.
An, S., Ma, Z., Lin, Z., Zheng, N., and Lou, J.-G. Make your IIm fully utilize the context. arXiv preprint arXiv:2404. 168/1, 2024.
Bai, Y., Lv, X.,, Zhang, J., Lyu, H., Tang. J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L.,, Tang, J., Dong, Y., and Li, J. Longalign: A recipe for long con- text alignment of large language models. arXiv preprint 2024.
Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023a.
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint urXiv:2309. 2023b.
Acknowledgements
We want thank all the anonymous reviewers for their valuable comments. This work was supported by the Na- tional Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), the Young Elite Scientists Spon-
Chen, Z.,, Chen, Q. Qin, L., Guo, Q., Lv, H., Zou, Y., Che, W., Yan, H., Chen, K., and Lin, D. What are the essen- tial factors in crafting effective long context multi-hop instruction datasets? insights and best practices. arXiv preprint arXiv:2409.0 2024.

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint
Laban, P., Fabbri, A., Xiong, C., and Wu, C.-S. Summary of haystack: challenge to long-context LLMS and RAG systems. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pp- 9885- 9903, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10. 118653/v1/222/ emnlp-main. 55.n. URL https: http::/:/aaaaathhlorr... org / 2024. e.:.4.mmpppmmm .n..5//........
2018.
Computer, T. Redpajama: an open dataset for training large language models, 2023. URL htsps::tp//.. //githb. com/ htmm/toggtheccomptteeeo hatpeerrrccopttteee//aaaaaaaaaaa Data.
RRedPajaaaaaa
Dao, T. Flashattention-2: Faster attention with bet- parallelism and work partitioning. arXiv preprini arXiv:2307.08691, 2023.
Li, S., Yang, C., Cheng, Z., Liu, L., Yu, M., Yang, Y., and Lam, W. Large language models can self-improve in long-context reasoning. arXiv preprint arXiv:2411.00 1,08147, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788 2024.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods, arXiv preprint arXiv: 109.07958 2021.
Fu. Y., Ou, L., Chen, M., Wan, Y., Peng, H., and Khot, T. Chain-of of-thought hub: A continuous effort to measure large language models' reasoning performance. arXiv preprint ar arXiv:2305. 17306, 2023.
Liu, H., Zaharia, M., and Abbeel, P. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Represen- tations, 2024.
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng, H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10177, 2024.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. and Liu, Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2023.
gkamradt.
Limtest- Lmmtest-needeeinahaastta...
https:
//github. , www.gkaaraaeecccoooooooooo LLMTest_
NeedleInAHaystaad 2023.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint urXiv:2405. 14734, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint (rXiv:2009.03300 2020.
MiniMax, Li, A., Gong, B., Yang. B., Shan, B., Liu, C.,
Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E., Li, G., Zhang, G., Sun, H., Dong, H., Zhu, J., Zhuang, J., Song, J., Zhu, J., Han, J., Li, J., Xie, J., Xu, J., Yan, J., Zhang. K.. Xiao, K., Kang, K., Han, L., Wang, L., Yu, L., Feng. L.., Zheng, L., Chai, L., Xing, L., Ju, M., Chi, M., Zhang, M., Huang, P., Niu, P., Li. P., Zhao, P., Yang, Q., Xu, Q., Wang. Q., Wang. Q., Li, Q.. Leng, R., Shi, S., Yu, S., Li, S., Zhu, S., Huang, T., Liang, T.. Sun, W., Sun, W., Cheng, W., Li, W., Song, X., Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y., Yu, Y., Yang, Y., Li, Y., Huang. Y., Li, Y., Huang, Y., Xu, Y., Mao, Y., Li, Z., Li, Z., Tao, Z., Ying. Z., Cong, Z., Qin, Z., Fan, Z., Yu, Z., Jiang, Z., and Wu, Z. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https: / /ttp/vixvv/.////////////////aata org/ abs 2000/551.0000 08313.
Hong, J., Lee, N., and Thorne, J. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07661, 2024.
Hsieh, C.-P. Sun. S., Kriman, S., Acharya, S., Rekesh, D., Jia, F,, and Ginsburg, B. Ruler: What's the real context size of your long- contecontext language models? arXiv preprint arXiv:2404.066 06650,6644400 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 096865, 2021.
Jangg A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Ouyang, I., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 27744, 2022.
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z. Chang, C.- Y., Chen, H., and Hu, X. Llm maybe longlm: Self- extend IIm context window without tuning. arXiv preprint arXiv:2401.01325 2024.
10

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007) 2023a.
Shi, F,, Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pp. 31210 31227. PMLR, 2023.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Effi- cient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023b.
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Roformer, Y. L. Enhanced transformer with rotary position embed- ding., 2021. DOI: https://doi. org/10. arg/10.1016/; neucom, 2023.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables imput length extrapolation. arXiv preprint arXiv: /2409, 2021.
Tang, Z., Zhou, K., Li, J., Ji, B., Hou, J., and Zhang, M. L-citeeval: Do long-context models truly leverage context for responding? arXiv preprint arXiv:2410.02111 2024.
Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C.,, and Lillicrap, T. P. Compressive transformers for long- range sequence modelling. arXiv preprint, 2019a. URL ht https: /tappi///rxiv..../// wwg//bs/gaaeoeooeeeo.eo/.. abaaabsss/111/ 05507.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L.,, Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05533 2024.
Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. Compressive transformers for range sequence modelling. arXiv preprint arXiv:1911.05500 2019b.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint MXXiv2204.02000 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.099 0098888,888 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Mitoš, P. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
Ravaut, M., Sun, A., Chen, N., and Joty, S. On context utilization in summarization with large language models. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers). pp. 2764-2781, 2024.
Weng, Y., Han, M., He, H., Chang, X., and Zhuang, B. Longvim: Efficient long video understanding via large language models. In European Conference on Computer Vision, pp. 453-470. Springer, 2025.
Ruoss, A.. Delétang, G., Genewein, T., Grau-Moya, J., Csordás, R., Bennani, M., Legg, S., and Veness, J. Ran- domized positional encodings boost length generalization of transformers. arXiv preprini arXiv:2305. 16843, 2023.
Wu, W., Wang, Y., Fu, Y., Yue, X., Zhu, D., and Li, S. Long context alignment with short instructions and synthesized positions. arXiv preprint arXiv:2405.03939 2024a.
Saeidi, A., Verma, S., RRV, A., and Baral, C. Triple pref- erence optimization: Achieving better alignment with less data single step optimization. arXiv preprint arXiv:2405. 2024.
Wu, W., Wang, Y., Xiao, G., Peng. H., and Fu, Re- trieval head mechanistically explains long-context factu- ality. arXiv preprint arXiv:2404. /5544, 2024b.
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L.,, Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long- scaling of founda- tion models. arXiv preprint arXiv:2309.1603: 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777 2017.
Shaw, P. Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv: 1803. 155, 2018
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of llm performance

---

LOGO-Long Long cOntext aliGnment via efficient preference Optimization
in machine translation. arXiv preprint arXiv:2401.08417,
2024.
Yang, A., Yang, B., Hui. B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang. F., Dong, G., Wei, H., Lin,
H., Tang, J., Wang, J., Yang. J. Tu, Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y. Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint ar arXiv:2407. 10671, 2024.
Yang. S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient train- ing. arXiv preprint arXiv:2312.066333 2023.
You, W., Tang, Z., Li, J., Yao, L., and Zhang, M. Revealing and mitigating the local pattern shortcuts of mamba. arXiv preprint 4rXiv:2410.15678, 2024.
Yu, T., Xu, A., and Akkiraju, R. In defense of rag in the cra of long-context language models. arXiv preprint arXiv:2409.01666. 2024.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, H. Sinklora:
Enhanced efficiency and chat capa- bilities for long-context large language models. arXiv preprint wrXiv:2406.05678, 2024.
Zhang, J., Bai, Y., Lv, X., Gu, W., Liu, D., Zou, M.. Cao, S., Hou, I.., Dong, Y., Feng, I.,, and Li, Longeite: Enabling Ilms to generate fine- grained citations in long-context qa. arXiv preprint wwXiv:2409.02899 2024a.
Zhang, P., Shao, N., Liu, Z., Xiao, S., Qian, H., Ye, Q., and Dou, Z. Extending Ilama-3's context ten-fold overnight. arXiv preprint arXiv:2404.1955: 2024b.
Zhu, D., Yang, N., Wang, L.,, Song, Y., Wu, W., Wei, F. and Li. S. Pose: Efficient context window extension of Ilms via positional skip-wise training. arXiv preprint a3Xiv:2309. 10400 2023.
Zhu, Q. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma. S., al. Deepseek-coder- v2: Breaking the barrier closed-source models in code intelligence, arXiv preprint arXiv:2406.1 //931, 2024.
12

---

LOGO Long cOntext aliGnment via efficient preference Optimization
A. Details of Preliminary Experiments
In this section, we illustrate the details of the preliminary studies section 1, including the experimental settings, task definitions, and the retrieval score calculation.
Experimental Settings In figure ((a) and figure 1(b), evaluate the model performance on the subsets in LongBench (Bai et al., 2023), including Single Document QA. Multi-Document QA, Summarization, and Few-shot tasks. For each long- context model, we utilize the same official instructions to guide the model prediction.
Multi-values Neelle-in-a-Haystaah In figure 1(c), we calculate the retrieval score on the Multi-values Needle-in-a- Haystack dataset, which requires LCMs to recall multiple values within the context. We provide an example in figure
Multi- Needle-ing in-a-nn----a Haystack
Context:
... context... ...
The best thing to do in San Francisco is to cat sandwich and sit in Dolores Park. context...
The best thing to do in New York is to eat a sandwich and visit the Statue of Liberty. context...
Question:
What the single best thing do in both San Francisco and New York?
Ground Truth: (preference) eat sandwich
Figure Demonstration of Multi-values Needle-in--a-aasttcc testing sample.
The formal definition of the task is follows: Given n questions eq and its corresponding answers K {vky (the needle), we insert K in synthetic context (the haystack) at random position index ranges / {up; Y We then require the models to answer based on the haystack with the inserted needle. It is worth noting that and K are unique and irrelevant to the context, ensuring that if an answer is correctly generated, it is indeed copied from the context, not from the model's internal knowledge.
Calculation of Retrieval Score Based on Wu al. (2024b), we define the retrieval score as the recall score of salient tokens located by retrieval heads. To enhance comprehension, we manage utilize familiar symbols and definitions that align closely with previous research. Specifically, denote the current token being generated during the auto regreseereesive decoding process as x, and the attention score of head as Rlel, In the task of Multi-values Needle-in-a-Haystedl an attention head h is denoted as retrieval head if it meets the following criteria:
x. E kis. where k; E K and x is token within any one of the needle sentences in K.
Cj = x, = arg max(a). j E p. P: E P. i... the input token that receives the highest attention probability by this head is token within any one of the needle K and is the same token as the currently generated token.
Let gh be the set containing all copy tokens (also can be viewed as the located tokens) and pasted by given head h, we define:
Retrieval score for head h [gh 0 ki
[kg
It is worth noting that the retrieval score represents token-level recall rate of the most attended tokens by an attention head. After obtaining the retrieval score for each head, we start by filtering out the non-retrieval heads by setting the threshold at 0.1. This means that if head performs copy-paste 10% of the time or more, it will be considered a retrieval head. Then, we caleulate the retrieval head score by averaging the scores of the top 10 attention heads from the remaining retrieval heads.
13

---

LOGO - Long cOntext aliGmment via efficient preference Optimization
Critical Context
Q: What are the occupations of Rachel and Monica in Friends currently ?
Distracting
[Entity [Entity
Irrelevant Context
Long Context Model
Pred: Rachel
waitress;
Monica is chei
[Entity_ 1]
(nnser__)) 1]
Correct
[Entity 21
[nnswr
Mitaiggmmett (Instuction Unfollow)
Pred: Jackey likes swimming
[Entity_
[Answer_I]
[Entity 2
[Anser
Misaligument
(Hallacination)
Pred: Rachel
Monica teacher
[Entity_1]
[Anserr_]]
[Entity_2]
Ansswrr 2
Misaligament (Halluciaaion)
Pred: Rachel is paleontologist Monica is a chef
[Entity_]]
[Anserr_ 1]
[Entity2]
Answer 2]
G waitress in coffee and Monica chef
Friends is show
Rachel works
Rachel paleantologist year ago 12
[Entiky_
[Answer_AA]
[Entity [Answer_2
Long Context (> 32K)
Statistic of Error Patterns
Statistic Error Patterns
18.00 dH2,
16
16. 0%
19.0%
9.0%
0%
34, 34.0%
Single Document
0%
Multi Document QA
Synthetic Task
Single Document QA
37
Multi Document QA
Synthetic
0%
Task
43.0%
1.-nnstuet (128K Window)
Mistral-instrruct-B.. (32K Context Window)
Figure 7. Demonstration and statistical analysis of different error patterns in long context tasks, where we have the following definitions of misalignment: Instruction Unfollow: The entities in the model's prediction different from the entities in the question; (2) Hallucination:
The entities in the prediction overlaps with the entities in the question, but the answer is incorrect.
B. Design of LOGO Training Objective and Error Pattern Definition in LCMs
Misaligned predictions generated from LCMs can be specifically categorized into two types: failing to follow instructions and generating hallucinations. In figure 7, illustrate these error patterns. Specifically, we define different patterns by utilizing the degree of overlap between entities in the responses and the questions, along with specific templates:
Instruction Unfollow: the entities the model's responses do not overlap with the entities the question.
Hallucination: there is partial intersection of entities between the model's responses and the question, and the entities in the response coincide with the main subject of the question.
worth mentioning that merely utilizing Named Entity Recognition (NER) models and rule-based methods proves inadequate for identifying these patterns. Instead, a more robust evaluation involving strong LLMs such as GPT-4 or human assessment required to accurately identify these patterns. Consequently, in the design of the LOGO training objective, we do confine to constructing cases with specific error patterns. Therefore, instead of finding one rejected sample with expand the rejected space push the model away from range of possible rejected samples.
specific error pattern,
C. Error Bound Analysis
In this section, we analyze the error bound of the LOGO loss function (equation 4):
1 M
log no (v) (1) -7)) -7)
B
LLOGO(TO) (r. an)
log
log (V(Wll>)
|ywl
M [vl
Explanation of the Components in LOGO
Sigmoid Function: The sigmoid function (22) = Ite maps vellee inputs to the range (0, 1), and the log-sigmoid function log (z) encourages large positive values of
14

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Model Probabilities: no (a) is the probability of the preferred response given input x, and (o (a:) is the probability of the dis-preferred response.
(1(())))
and M aim to normalize the oog-rrbbabilities by the lengths of the responses to account for varying lengths and scale the contribution using B. is a margin prerrpaaameter to ensure that the difference between preferred and dis-preferred responses exceeds a certain threshold.
Scaling Factors bull
The LOGO function aims to maximize the difference between the (normalized) log- pyg-prbbabilities of the preferred response and the average of the dis- diseerefe responses beyond margin 7. Specifically, it encourages:
M
log To (Bwlx)
log To (3t ( xx) 2 7
M 13
To analyze the error bound, we can proceed by considering the following steps:
1. Bounding the Loss Function
Upper Bound:
Since log (z) < 0 for all real the negative log-sigmoid loss - log 2 0.
Lower Bound: The function - log (2) increases without bound as z + -oo, leading to potentially infinite loss
values. However, in practice, model probabilities (y(ar)) are bounded below by small positive value due to numerical stability (c.g., using softmax outputs and adding small E).
Assuming Bounded Log-Probabilities
Let's assume that there exists a constant C' such that:
log <)(()>) S C, Vy, 2, 0.
This assumption is reasonable since (y(yx 0 for numerical stability.
3. Bounding Given the boundedness of log (y|xr)
8
BC y/
log To (Bw
y/
M
,3
30
log To (v)
(x)
S
M 1
leu
Thus, is bounded:
C
|2|
S
3
+
+
lrl.
yaol
Lipschitz Continuity of the Loss Function
The function
log (2)) Lipschitz continuous with Lipschitz constant
since:
e= dz - (- = Ite- 1te S 2' V R.
d
5. Applying Concentration Inequalities Since the loss function is Lipschitz continuous and the losses bounded, can apply concentration incquality, i.c., McDiarmid's Inequality, bound the difference between the empirical loss and the expected loss.
Let {(I+, Mw Vi,
(1--MM) (.).A ) be N i.idd samples. Define the empirical loss:
(LoGO(TO) = N - log a (2+),
15

---

LOGO Long cOntext aliGnment via efficient preference Optimization
where 2, the z corresponding to the i-th sample.
McDiarmid's Inequality states that for all > O:
2
(CLoco(TT) LLOGO(TO) 26) I exp
Σ o
where e, is the maximum change in the loss function due to the replacement of the i-th sample.
6. Determining the Bounded Differences the Lipschitz continuity:
Since the loss function change is bounded due to the boundedness of a and
1 G = Fnew,i Fpld,i
where Anew,i and i are the values of z before and after the change in the i-th sample. Given the boundedness of log ne (ylar (z)) and the response lengths, we have finite Cy.
7. Bounding the Generalization Error Using the inequality, we can bound the probability that the empirical loss deviates from the expected loss by more than
P ((LLoco(TT) (To) 2x) < 2exp
-2 Σi c.
This inequality provides high-probbbbility bound on the generalization error error-the difference between the expected loss and the empirical loss decreases as N increases.
8. Error Bound in Terms of Sample Size and Variability
The error bound depends on:
Sample Size N: Larger N leads to tighter bounds.
Variability c: Smaller c, (less variability the loss function) leads to tighter bounds.
Takeaway Based on the above analysis, we can get:
To achieve a small generalization error, we need a sufficiently large sample size N. In this paper, there are 0.3B tokens (6,000 samples) for training, which is enough for convergence.
Ensuring that the model probabilities ()(y)>)) (z) do assign extremely low probabilities (avoiding numerical instabilities) to keep the loss function bounded and the e; small. This is achieved by adopting the strong LLMs (e.g., Llama-3) for training.
This analysis assures that with sufficient data and proper control of the model probabilities and response lengths, the loss function (To) will have small generalization error, leading to reliable model performance unseen data.
D. Convergence Property of LOGO from Gradient Analysis Perspective
In this section, we analyze the convergence property of the LOGO training objective from the gradient analysis perspective.
16

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
D.I. Gradient Analysis
We first compare the gradient among three training objectives, i.e., DPO, SimPO, and LOGO. The gradient of those three training objectives can be written as:
V (OLOOOTTT) -9 (a (1 D
dg
V log The (Vw
o log wo (xr)
increase likelihoud on Mie
decrease Likellhood on
VoLsimo((TT) (a (1m
Vo log (Bhrl
Vo log To (3rl (x)
(6)
increase likelituud on Mhe
decruuse likelitoud on
(LoCo(TT) = -E, (anw My
(1..M) M.nD Lo
log To (tw (x)
log To (v)
Mho
M Isn
likelibood the
decrense likelthood on v1.
Charrs .633
where
To To(
( (z))
da
[log
B log
Teet (ee (x)
Trer
99
log ne (vila)
B
log (ywl())) +
ywll
la
B
log ne (M)
00 (c)
B Null
log no (2)) +7)
M [d
In terms of gradient weight computation, SimPO and LOGO are similar in that they do not rely on a reference model. Instead, both methods caleulate gradient weights based the policy model itself. On the one hand, for both SimPO and LOGO training objectives, weight 8e is higher for samples where the model incorrectly assigns higher likelihood to the dis-preferred output y or yi (I--M) thereby focusing on correcting the model's mistakes. On the other hand, by considering multiple negative samples vt (1.....) LOGO enriches the learning signal of rejected samples. This approach allows the model to learn more comprehensive representation of undesirable outputs, improving its ability to reject a broader range of negative samples and helping the model to learn more patterns.
D.2. Convergence Properties
The combination of self-contained gradient weights and length normalization in LOGO promotes stable convergence. Since la focuses on the policy model's own mispredictions without relying on reference model, the learning process can adapt more freely based on the actual data, potentially leading to faster and more robust convergence. Besides, the use of a logistic loss function with a margin parameter y introduces non-linearity to the optimization problem. While the inclusion
of multiple negative samples (1) can provide richer learning signal, and the length normalization helps in maintaining balanced updates, which can aid in convergence.
17

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Positional Indices Synthesia
Batch Samples
Posstoral Inddo [0-19]
Training Input Samples
Acthoted you Pooaaa indio (0-
Posttionelndes Index (0-19)
Activetedd Pouttonet indes [u- 11-3
Povitional Indicas
Harge All
Poettional codet
[0-3|U]56-31]]
By
Aetivoeed Posttoral Indos
(0-31022-48) 24
Act vated Positional [0-43)
Activated Postiinall Indices
0-19 [16-31] 28-43) 0 0-43]
(a) Continuous Chunk Positional Indices Synthesis
Positional Indicos Synthesia
Model Perceptual Field
Batch Samples
Arthanedd fattunal
(L--JjiLL--- 19)- 22 40-03-4-3
Training Input Samples
Acthvated Indes
7 131020-23) JIRL 35/J/
Autisfeed Pojentanaa Index
(8-77]))6-199U 28-33ju-ee (8-33]U.40-55)
Arthoated tout indes (a ca--jll 151
Poultoona Indicns
Morge All
Accifpan Popit ond Inderr
(24- 2771128--0
(0- 20-3) U3 60-43)
Aoti Index
Activated Positional Index 0-433
Activated Fasibonal Indices
(16-19) 19 (36
(b) Sparse Chunk Positional Indices Synthesis
Figure 8, Illustration of positional indices synthesis process, where the original context length is 19, and we extend it to a context length of 43. is noteworthy that one batch in the figure corresponds training sample of LOGO, which contains preference instance and several rejected samples.
18

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
E. Positional Indices Synthesis
E.1. Theoretical Guarantee of Positional Indices Synthesis
The theoretical guarantees for the position synthesis method can be equated to understanding why Relative Position Encodings (RPE) are effective. Existing works (Shaw et al., 2018; Zhu et al., 2023; Wu et al., 2024a) have already demonstrated that using positional indices synthesis can significantly improve training efficiency. Simply put, for the Transformer model, the model learns (j i), i.e., the difference between positional indices, understand the relative position information between any two tokens. The context window scaling can be interpreted the model's tendency to learn information about larger (j - 2) values. Referring to the mathematical formulation of RoPE, the attention mechanism can be expressed as follows:
Atinay = (QKT + +((I.5) V
Vd
where Q, K, and V are the query, key, and value hidden states, respectively.
To perform the length extrapolation, one only needs to consider how to make i) larger. The conventional approach is to increase the sequence length to enlarge the value of (j - i), with each token corresponding to an absolute and j value. As for position synthesis, it only needs to consider changing the values of and j without altering the actual sequence length.
potential issue is that some position indices may missing in the positional synthesis method, and we illustrate how we compensate for the missing relative positions in the below section.
E.2. Implementation Details of Positional Indices Synthesis
We visualize the positional indices synthesis process figure Specifically, to ensure that the synthesized positional indices do not disrupt the original text's semantic structure while maximizing the extended context size, we employ two different strategies for positional bias B: Continuous Chunk Positional Indices Synthesis (figure 8(a)) and Sparse Chunk Positional Indices Synthesis (figure 8(b)). For Continuous Chunk Positional Indices Synthesis, the positional bias within the same chunk is consistent. For instance, in the first chunk Co. the positional bias (do, b,, .., bicc), are the same value sampled from distribution (11, (K k)/N).
This ensures that the semantic structure within the chunk remains intact but can lead to sparse synthesized posi- tional indices, as there will be significant gaps between the positional indices among different chunks. Thereby, we propose Sparse Chunk Positional Indices Synthesis to fill these gaps, where each positional bias b; is sam- pled uniformly according to Equ. 2. Considering that Sparse Chunk Positional Indices Synthesis might disrupt the semantic structure of the text, we set the ratio of data for Continuous Chunk Positional Indices Synthesis and Sparse Chunk Positional Indices Synthesis to 9:1 in actual deployment.
>100
PPL explosion
PPL explosion
PPL explosion
12
10
F. LOGO Training and Evaluation Details
To accelerate the training process and save GPU memory, we adopt DeepSpeed Zero 3 (Aminabadi al., 2022). All the experiments are conducted on 8x A800 (80GB) GPU machine, and all the training experiments are completed within 16 hours. We train all the models for two epochs, amounting to total throughput of 0.3B 2 0.6B tokens. The best model checkpoint is then selected based on per- formance on the validation set. For the training of the baseline, we followed the corresponding baseline settings.
Llame 3-88-Instruct 80K
Mistral-Instrutt
wttp:rcttt.2220.233222223 (LOGO)
Llame-3-08- 3-i---------n-ttt----- 8OK (LOGO)
Lame-2-78-Chat-e
Samma---B-Instrucc
Lamma-2-7--Chat--- (LOGO)
2amma-2---insttuutt-nn (LOGO)
1amaa---8-instruct--t
MistraHnntmutt322400222
Llama-3-88-iree -amma-3-8--nsstruct-aKKK (LOGO)
2K4K 8K
16K
32K
64K
Figure 9, Evaluation results of language modeling task. The solid and dashed curves represent the PPL of the basclines and LOGO.
19

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Baseline LOGO
Instruct-Tuning
Long-context Scaling of SCMs (x8)
Long-context Alighment of LCMs
LOGO
MMLU
TruthfulQA
ARC (Avg.)
65
70
65
"alignment from
60
60
LOGO
55
55
50
50
45
45
40
40
Llama- aama-3-800. 3-80-
Llama Llama-2-778 2-78-
Llama-3-an 8-80-
Llama-2 2-B--
Mistral- 78-
Llama-3 3-BB-
Llama-2 7B-
Llama- 3-8B-
Llama 2-7B-
Mistral- 7B-
Liama- Alama-3-8B 3-88-
Llama-2: 2-78-
Llama- 3-88-
Llama 2-7B-
Mistral: 78-
Instruct (8K)
Chat (4K)
Instruct (B0K)
Instruct (80K)
v0.2 (32K)
Instruct (8K)
Chat (4K)
Instruct (80K)
Instruct (80K)
v0.2 (32K)
Instruct (8K)
Chat (4K)
Instruct [BOK]
Instruct (80K)
vo.2 (32K)
Figure 10. Model performance on short-context tasks, including MMLU, TruthfulQA, ARC.
When testing on LongBench, we used the officially recommended context length of 32K (Bai et al., 2023), truncating any input sequence that exceeds this length.
G. More Experimental Results of LOGO
G.1. Evaluation Results on Language Modeling Task
We test the language modeling capability of LCMs by calculating the Perplexity (PPL) on the Gutenberg (PG-19) testing set (Rae et al., 2019b), with context lengths ranging from 2K to 64K. Considering that extremely long context lengths can cause the PPL caleulation to exceed GPU memory, we apply the sliding window approach proposed by Press et al. (2021). As depicted in figure 9, for LCMs, such compromise the language modeling capability since the solid line (PPL of the backbone model) and the dashed line (PPL of LOGO) almost completely overlap. In the case of SCMs, such the Llama-3-8B-Instruue effectively scales the context window size of baseline models (the purple dotted curve versus the purple solid curve) but also achieves lower PPL score compared to the SFT method since the yellow dotted curve (PPL of much lower than the blue solid curve (PPL of Llama-3-88 S--Innsrrcct-OOE
and Sauma-2-7BInnstruett
using LOGO does
model, LOGO not only
G.2. Evaluation Results on Short-context tasks
Apart from the model results on MMLU (Hendrycks et al., 2020) and TruthfulQA (Lin et al., 2021), we also experiment with ARC (Hard and Easy) (Clark et al., 2018). The model performance shown in figure 10, which consistent with the conclusions presented in the main text.
G.3. Scaling to Longer Context Window Size
We scale the context window size of the performance Table 2. Specifically, for longer context lengths, using only 0.3B tokens is insufficient to cover all position indices. Therefore, we expand the dataset size as the context length increases. We can observe that as scale the context window size with more data, the model's performance also improves.
model from 80K to 256K and report the model
H. Comparison between SFT and LOGO
As shown in figure 11, we illustrate the impact of SFT (with two loss calculation strategies following (Xiong et al., 2023)) and LOGO on the model's generation and understanding performance throughout the training process. We plot the trends of retrieval score (understanding ability) and recall score (generation ability) along the training progress. We can observe that
20

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Table
Model performance when scaling longer context window size.
Model
Tokens
S-Doc QA
M-Doc QA
Summ
Few-shot
Synthetic
Avg.
Yi-6B-200K
39.1
25.1
33.8
25.6
56.6
36.0
-lama-3-8B Instrrct-LOG----55
0.3B
44.0
41.2
28.1
68.6
53.0
47.0
Llama-3-4 3-8B- 1-8B--nstrruc-Lee Instruct-LLGG-1222 --8K
1.2B
43.8
40.9
28.0
68.6
52.6
46.8
Llama-3-8B--ne 3-88--Innttru---000-----0
4.8B
44,9
42.6
29.8
69.4
53.9
48.1
Model Performance wtih Long Instruction Tuning (SFT)
Model Performance with LOGO
Heal Training Length: 50K; Testing Length: 64K
Real Truining Length: 8K; Testing Length: 64K
3.0
0.20
1.0
0.20
Score
Retrival Score
0.8
-0.36
0.8
0.16
Performance
0.6
-0.12
0.4
00008
0.4
00.08
0.4
0.08
0.04
0.0
200
400
630
000
1000
0.00
0.0
200
400
600
800
2000
0.00
0.0
200
400
600
800
1030
0.00
Training Steps
Training Steps
Training Steps
(a) Context Loss
Prodiction Loss
(b) Prediction Loss
(c) LOGO Loss
Figure 11. Comparison between SFT and LOGO training strategics on the synthetic retrieval task.
applying SFT loss to the entire sequence leads to a gradual decline in the LCM's understanding ability, accompanied by performance fluctuations; while applying SFT loss solely to the prediction portion shows no significant improvement in model performance. Nevertheless, applying LOGO can steer LCMs away from misaligned samples, thereby enhancing the recall score. Simultaneously, it improves comprehension abilities, enabling the model to retrieve more key information within the context.
I. Trade-off between Training Computation and Model Performance
In this section, we present the trade-offs between computational efficiency and performance gains across different long- context alignment strategies. Specifically, we employed ring attention (Liu et al., 2024), a context-paralle approach, to implement training strategies for longer imput sequences. We compared the following four strategies: (1) direct SFT training with the maximum input length supported by 80GB GPU memory (64K); (2) SFT combined with ring attention train on longer sequences (128K); (3) our LOGO method; and (4) the LOGO method extended longer training sequences in combination with ring attention. As shown in Table 3, we observe that extending the context length during training leads to slight performance improvements but comes at the cost of longer training times and reduced data throughput. For example, compared to LOGO, LOGO with Ring Attention achieves a 0.9-point improvement in performance. However, the throughput on GPUs only half of the original training setup (since we set the context- -artaxtt size as 2), resulting in a total training time of 30 hours. Thereby, we believe that utilizing a position synthesis strategy process more diverse data within 30 hours would likely yield better results while maintaining more affordable resource consumption.
J. Case Study of LOGO Data
In this section, we provide the training samples built based on the LOGO training data construction pipeline as illustrated in Sec. 3.2.1. We present the training samples in figure 12, figure 13, figure 14, and figure 15, where the training data exhibits different error patterns (misalignments) in their rejected samples.
21

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Table
Training resource statistics among different settings. For ring attention (denote with *), context parallel size i.e.,
every 2 GPUs share I training sample.
Total Throughput
Actual Training
Training Time
Real-world Task
Language Modeling
Training Strategy
Memory/GPU
Bsz/GPU
(8 GPUs)
Length
(2,000 steps)
SFT SFT Ring Attention
80GB
samples
64K
14h
43.2
6.6
45GB
samples*
128K
24h
44.3
6.6
LOGO Ring Attention
69GB
3
12 samples"
64K
30h
47.7
9.8
LOGO
64GB
3
24 samples
12K
16h
47.0
10.4
Case Study
Context:
context...
If class time used to allow students to apply what they have learned cooperation with their peers, then the Flipped Classroom Model encourages collaboration, inquiry, and problem solving.
context....
Question:
Why is it important for teachers to use student-centered activities in the flipped classroom model?
Ground Truth:
(preference)
To encourage communication, inquiry, research, problem solving, and collaboration.
Wrong PredictionI:
(Instruction Unfollowing)
How can teachers create student-centered learning experiences in the classroom? What are some possible challenges in creating student-centered learning experiences the classroom?
Wrong Prediction2: (Hallucination)
Blended learning in action: A practical guide toward sustainable change. Corwin, Please note that you need to have at least basic subscription to access the full text of the references...
Figure 12. LOGO training samples: casel.
K. Error Analysis of Model Response
We conduct an errоr analysis on the results generated from different models, as illustrated in Figures 16~ 20. Specifically, we focus comparing the generated results from three models: Llama-2-LOGO, PoSE (Zhu al., 2023), and LongAlign (Bai al., 2024), all of which commenced training based the Llama-2-Chat model (Touvron et al., 2023). As demonstrated in the cases, we mark irrelevant content with wavy lines and relevant content with underlines. Our analysis reveals that LOGO can generate accurate responses without being influenced by distracting information. In contrast, the other two methods (PoSE and LongAlign) are susceptible to interference from irrelevant information, which leads to wrong outputs.
22

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Case Study 2
Context:
context...
As always in quantum physics, the joint measurement on Alice's two photons, concerning their relative state, produces a truly random result, just one among several possibilities.
... context... ...
Alice's s joint measurement thus produces random result. Depending this result, Bob's photon will always produce the result that the initial photon would have produced if it had been measured in the same direction, else it will produce the opposite result to the one that would have been produced by the initial photon. context...
Question:
What the significance of the energy correlation between the two infrared photons?
Ground Truth: (preference)
If one photon has energy above the average, the other photon will necessarily have an energy below the average.
Wrong PredictionI:
(Instruction Unfollowing)
Bell'' game is theoretical game where two players, Alice and Bob, make independent choices of the polarization state of two photons before measuring the correlations between their polarizations..
Wrong Prediction2: (Instruction Unfollowing)
None, since there is no mention of energy correlation between two infrared photons the provided context. The context only discusses quantum nonlocality, Bell game, and quantum teleportation...
Figure LOGO training samples: case2.
Case Study
Context:
context...
What the slope of the line containing points (2, -3) and (4, 3)? slope 3 context... ...
A slope that goes from low to high is positive. A slope that goes from high to low is negative. A slope that goes straight across is O.
context...
Question:
What the slope of the line containing points (2, -3) and (4, 3)?
Ground Truth: (preference)
Wrong PredictionI:
(Instruction Unfollowing)
Do you want solve it step by step?
Wrong Prediction2: (Hallucination)
The question can be interpreted as the difference in y-values divided by the difference in x-values. That is, let yl = and y2 = 3. Then xl = 2 and x2 = 4. The slope is: =
Figure 14. LOGO training samples: case3.
23

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Case Study 4
Context:
context...
And then there was no more speaking of the old lady, the house with the candle and the woman who was like a candle until, the last train, they were nearly home. context...
Question:
Why does the narrator stop asking his father about the grapes and the grape-vine?
Ground Truth: (preference)
Because of the unpleasant encounter with the old lady at the house.
Wrong PredictionI: (Hallucination)
The narrator stops asking his father about the grapes and the grape-vine because his father is not responding to the questions. The narrator is trying to learn about the grape-vine and its whereabouts but his father is not willing to provide any information...
Wrong Prediction2: (Hallucination)
Blended learning in action: Because his fathers eyes seemed to fill with water and he began speak in small voice. At the same time, the narrator himself felt like his heart had been "tom in half."...
Figure 15. LOGO training samples: case4.
Error Analysis 1
Context:
context...
AEM outpoerforms both LEM and DPEMM by 6.5 and 1.7 respectively in F-measure on the FSD dataset, and 4.4 and 3.7 in F-measure on the Twitter dataset. We can also observe that apart from K-means, all the approaches perform worse on the Twitter dataset compared to FSD, possibly due to the limited size of the Twitter dataset. context... ...
Question: What baseline approaches does this approach out ttrfperfomm?
Ground Truth: K-means, LEM, DPEMM.
LOGO (Ours): AEM outperforms both LEM and DPEMM.
LongAlign: The proposed approach outperforms the baseline approaches all three datasets.
PoSE-YaRN-96k: LEM and DPEMM.
Figure Error analysis responses from different models.
Error Analysis 2
Context:
context...
Larkin I. Smith Larkin Irvin Smith (June 26, 1944 August 13, 1989) was an American Congressman from Mississippi serving for seven months until he was killed in a plane crash in Perry County, Mississippi in 1989. Smith was born in Poplarville, Mississippi Nona Orene Bounds and her husband Hezekiah K. Smith.
... context...
Question: In which county was Larkin I. Smith born?
Ground Truth: Pearl River County, Mississippi.
LOGO (Ours): Pearl River County, Mississippi.
LongAlign: Poplarville, Mississippi.
AASE-YaRN99kk: Pearl River County.
Figure 17, Error analysis responses from different models.
24

---

LOGO - Long cOntext aliGnment via efficient preference Optimization
Eiror Analysis 3
Context:
context...
Locked On (novel) Locked On a techno-thriller novel written by Tom Clancy and Mark Greaney released on December 13, 2011.
context...
In 1832, Fougeres lived in the de Navarin.
context....
Question: What type of media does Locked On and Tom Clancy have in common?
Ground Truth: Novel.
LOGO (Ours): Novel.
LongAlign: Book.
PoSE- : Video games. Books. Film.
Figure Error analysis responses from different models.
Error Analysis 4
Context:
context...
Retired merchants; just now in love with art; have country-house Ville d' Avray, and ten or twelve thousand francs year.At the third sitting pere Vervelle mentioned fine gallery of pictures which he had in his country-hour at Ville d' Avray-Rubens.
context...
Question:
Where do Mr. and Mrs. Vervelle live?
Ground Truth: In a mansion in Ville-d' Avray.
LOGO (Ours): Ville d' Avray.
LongAlign: the country.
PoSE-YaRN-96k. They live in the rue de Navarin.
Figure 19, Error analysis responses from different models.
Error Analysis 5
Context:
context
Thaddeus P. At the time of his death, he was also the last surviving of the eminent surgeon Valentine Moot...Upon his death in 1865, Mott was interred at Green-Wood Cemetery in Brooklyn, New York.
context...
Beaulieu-sur-Lorre (French promunciation:liteaal)) Beaulieu on Loire) is a commune the Loiret department north- central France.
Two days later, Anthony Roberts was on the scene with detachment of Philadelphia police.
... context...
Question:
Where was the place of death of Thaddeus P. Mott's father?
Ground Truth: New York.
LOGO (Ours): New York.
LongAlign: Beaulieu-sur-Loire.
PoSE- DOSE-YaRN-96k: Anthony Roberts.
Figure 20. Error analysis of responses from different models.
25


Reviews:
[{"review_id": "JGlh59nhTY", "reviewer": "Reviewer_icM3", "summary": "The paper addresses the issue of long-context models struggling with generating coherent and accurate responses in real-world tasks.\nIt proposes LOGO, a preference optimization-based training strategy for long-context alignment, which includes efficient preference data synthesis and a reference-free training objective.\nExperiments demonstrate improvements in multiple long-context tasks, with LOGO outperforming existing methods and maintaining or improving performance on short-context tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer icM3, thanks for your insightful comments and suggestions. Below is our detailed response:\n\n------\n\n**[Concern 1]** Rigor of the main claim that increasing training data generally improves model effectiveness\n\n**[Re]** We acknowledge that increasing high-quality training data can improve model performance, but training efficiency is a critical factor. Our work focuses on the efficient long-context alignment algorithm in LOGO rather than pure data scaling. Indiscriminate pure data scaling offers limited returns on long-context tasks. As shown in Table 1, SFT with 5B tokens (Data-Engineering-80K) improves LongBench performance by only +1.3 points, whereas LOGO achieves +2.5 points using just 0.3B tokens (16× less data). Additionally, Table 3 shows that LOGO improves performance (40.7 → 47.0), while SFT obtains a smaller gain (40.7 → 43.2) with a similar amount of data. Moreover, scaling data size in SFT requires careful balancing, as improper ratios (Figure 10) can hinder alignment.  `We also compare LOGO with other long-context DPO method, please refer to the second point of Reviewer 6VnN.`\n\n------\n\n**[Concern 2]**  Evaluation results on LongBench-V2\n\n**[Re]** We have included the results in LongBench-v2 below. \n\n| Model                                | Overall        | Easy        | Hard        | Short        | Medium       | Long        |  \n|--------------------------------------|--------------|------------|------------|------------|------------|------------|  \n| Llama-3-8B-Instruct-80K-QLoRA-Merged | 10.3        | 9.4        | 10.9       | 11.1       | 11.2       | 7.4        |  \n| + LOGO                               | 18.3 (**+8.0**) | 17.2 (**+7.8**) | 19.0 (**+8.1**) | 20.6 (**+9.5**) | 15.3 (**+4.1**) | 26.9 (**+19.5**) |  \n| Mistral-7B-Instruct-v0.3             | 25.6        | 24.5       | 26.4       | 30.0       | 25.6       | 18.5       |  \n| + LOGO                               | 29.8 (**+4.2**) | 30.2 (**+5.7**) | 29.6 (**+3.2**) | 35.0 (**+5.0**) | 28.4 (**+2.8**) | 26.9 (**+8.4**) |  \n| Qwen2.5-7B-Instruct                  | 30.2        | 32.3       | 28.9       | 37.8       | 25.1       | 27.8       |  \n| + LOGO                               | 32.8 (**+2.6**) | 35.9 (**+3.6**) | 30.9 (**+2.0**) | 40.6 (**+2.8**) | 28.8 (**+3.7**) | 33.3 (**+5.5**) |  \n\nWe observed that Llama-3-8B-Instruct-80K-QLoRA-Merged performed significantly worse than expected, with results even lower than the *25% random guessing baseline*, while LOGO can improve by an average of 8 points. We have carefully verified this outcome using the official code and double-checked our implementation to ensure its correctness.  After applying **LOGO training**, both models exhibit substantial performance gains, further validating the effectiveness of our approach.  \n`Additionally, based on the suggestions from Reviewer 2obo, we have also provided the evaluation results on **Ruler** in the second point in the rebuttal box for Reviewer 2obo.`\n\n------\n\n**[Concern 3]** Innovative Contributions\n\n**[Re]** Long-context capabilities have become indispensable, making effective long-context alignment more critical than ever. To our knowledge, **LOGO is among the first open-source RL methods tailored for long-context alignment during the time of this paper submission**, whereas prior works primarily achieve alignment with SFT. As demonstrated earlier, SFT alone yields limited benefits for long-context tasks, and while RLHF is a promising alternative, existing RL methods (e.g., DPO) are not well-suited for long-context alignment due to deployment challenges and inefficiencies. \n\nWhile SimPO is effective for short-context generation, it lacks adaptations for long-context tasks. LOGO introduces: 1) **Positional synthesis**, enabling efficient handling of long sequences. 2) **A novel preference data construction method**, along with a novel preference strategy, to address the lack of long-context evaluation models. \n\n------\n\n**[Question 1]** Entity Extraction and Importance Scoring\n\n**[Re]**\n  1) **Entity Extraction**: As noted in Section 4.1 (Lines 261–262), we use spaCy to extract entities (e.g., person names, locations) from questions and chunks.\n  2) **Importance Metric**: In long-context QA tasks, questions often target specific snippets within the long context. Overlapping entities between a chunk and a question indicate higher relevance (e.g., a chunk mentioning \"Berlin\" is likely critical for a question about \"Germany’s capital\"). We empirically found that selecting top-K chunks (e.g., K=16 for 512-token chunks) suffices to cover salient information, balancing efficiency and accuracy. Therefore, we use the number of overlapping entities as a key metric to measure importance, and the importance score reflects the relevance of each chunk and the question.\n\n------\n  \nWe hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.", "claims_and_evidence": "The main claim of this paper that LCMs can achieve significant improvements in real-world tasks by training models with LOGO is not entirely rigorous, as an obvious issue is that increasing the relevant training data will always improve the model's effectiveness by training.", "methods_and_evaluation_criteria": "It is sense-making.", "theoretical_claims": "I have checked the proofs in this paper, which are reasonable. For example the proof that theoretical guarantee the synthetic positions can cover all possible scenarios listed in Appendix E.", "experimental_designs_or_analyses": "I have checked the soundness of the experiments in this paper. For example the evaluation on LongBench introduced in Sec. 4.2. I think it is valuable to evaluate the proposed model on the newly released version LongBench-v2.", "relation_to_broader_scientific_literature": "The proposed training objective can contribute to the community for improving models on references corpus.", "essential_references_not_discussed": "The references are comprehensive.", "other_strengths_and_weaknesses": "Strengths\n- I think the proposed training objective, which is similar to the SimPO, is useful to training LLMs with references optimization.\n- The experiments are extensive to investigate the effectiveness of the model on various kinds of tasks.\n\nWeakness\n- As I mentioned above, the innovative contributions of this paper are limited. The proposed training object for LOGO seems like a variant of the SimPO. Further, the data construction pipeline is a widely used workflow.", "other_comments_or_suggestions": "The captions in Figure 1 should incorporate the benchmark (i.e., MMLU) used in the evaluation.", "questions_for_authors": "- In the section of Importance Scoring with Automatic Evaluator, how do you extract entities in both question and context? Why do these entities mater for measuring the importance? Why do these entities mater for measuring the importance of chunks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGhM1OCOxm", "reviewer": "Reviewer_2obo", "summary": "The paper addresses the challenge that open-source long-context models (LCMs) struggle with generation quality in long-context tasks, despite having strong information retrieval capabilities. These models often produce misaligned results, such as hallucinations and instruction-following errors, leading to low recall scores.\nKey questions addressed:\nExisting approaches primarily focus on scaling context length by adding more training data at the supervised fine-tuning (SFT) stage, which mainly improves retrieval capabilities.\nThere is a significant gap between retrieval and generation capabilities in LCMs—while they can locate important information, they struggle to use it effectively.\nConstructing long-context preference pairs is difficult and underexplored in the literature.\nProposed approach:\n The authors propose LOGO, which consists of three key components:\nA long corpus is broken into chunks, and each chunk is evaluated based on the number of overlapping entities with the given question.\nPreferred and dispreferred samples are generated by combining different chunk compositions and prompting the model to produce responses.\nPosition indices synthesis is used during training to ensure efficiency and fit within hardware constraints.\nMain results:\nLOGO uses only 0.3B tokens.\nAchieves comparable performance to GPT-4 and LLaMA on LongBench.\nMaintains strong performance on standard benchmarks like MMLU.\nIncludes ablation studies on hyperparameters such as the number of negative samples and the impact of SFT regularization on final results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2obo, we sincerely appreciate your thorough review of our work and the detailed feedback provided!\n\n---\n\n**[Concern 1]** A fairer comparison of YaRN\n\n**[Re]** We have conducted the experiment and found that incorporating YaRN indeed leads to further performance improvements:\n\n| Model                  | S-Doc QA | M-Doc QA | Summ | Few-shot | Synthetic | Avg.  |\n|------------------------|---------|---------|------|----------|-----------|------|\n| Llama-3-8B-Ins-8K     | 38.0    | 36.6    | 27.4 | 61.7     | 40.9      | 40.9 |\n| + YaRN-64K           | 39.8    | 36.7    | 28.8 | 65.4     | 49.0      | 43.9 |\n| + LOGO-64K           | 39.8    | 36.7    | 28.8 | 65.4     | 49.0      | 43.9 |\n| + LOGO + YaRN-64K    | **40.7** | **37.4** | **28.9** | **67.3** | **50.4** | **44.9** |\n\n---\n\n**[Concern 2]** Adding evaluations with full-context retention \n\n**[Re]** We have conducted the experiments and provided the evaluation results on Ruler below.\n\n| Model                                | Length | niah_1-3 | niah_multikey_1-3 | niah_multivalue | niah_multiquery | vt    | cwe   | fwe   | qa_1-2 | Total |\n|--------------------------------------|--------|----------|-------------------|----------------|----------------|-------|-------|-------|--------|-------|\n| Llama-3-8B-Instruct-80K-QLoRA-Merged | 32K    | 100.0    | 99.7              | 87.2          | 87.2          | 94.6  | 27.7 | 91.9 | 62.3   | 81.3  |\n| + LOGO                                 | 32K    | 100.0    | 100.0             | 93.2          | 92.5          | 95.2  | 28.1 | 93.2 | 66.2   | 83.5 **(+ 2.2)**  |\n| Llama-3-8B-Instruct-80K-QLoRA-Merged | 64K    | 100.0    | 99.0              | 84.5           | 84.5           | 92.6 | 0.2  | 78.7 | 59.5   | 74.9  |\n| + LOGO                                 | 64K    | 100.0    | 100.0             | 90.6          | 88.2          | 93.5 | 1.5  | 86.2 | 62.8   | 77.8 **(+ 2.9)** |\n| Llama-3-8B-Instruct-80K-QLoRA-Merged | 128K   | 99.9     | 77.9              | 76.0          | 76.0          | 88.6 | 0.3  | 81.9 | 52.0   | 69.1  |\n| + LOGO                                 | 128K   | 100.0    | 93.8              | 81.2           | 79.2          | 89.6  | 1.3  | 82.2 | 58.4   | 73.2 **(+ 4.1)** |\n\n---\n\n**[Concern 3]** Lack of training process and evaluation details\n\n**[Re]** Below, we provide more details:\n - **Fairness of Comparisons in Training Competitor Models** We acknowledge the token count difference between Data-Engineering-80K (5B tokens) and LOGO. For evaluation, we used the open-source pretrained checkpoint of the Data-Engineering-80K model, but the token format varies from LOGO, complicating direct comparison. Nonetheless, LOGO outperforms Data-Engineering-80K despite using a smaller dataset (0.3B tokens), showing its efficiency.\n- **Hyperparameters and Negative Samples** We used a fixed learning rate of 5e-5, with a cosine decay scheduler (100 steps warmup from 1e-8). The training setup included 8× A800 GPUs, global batch size of 64, microbatch size of 4 per GPU, and gradient accumulation of 2. The impact of negative prompt numbers is discussed in Section 5.1, with additional details in Appendix F.\n- **Reliability of Perplexity** Perplexity may not fully reflect long-context performance. We placed the perplexity results in Appendix G, using the PG-19 dataset as a test set, not for validation during training.\n- **Inverse Trend of Perplexity and Performance** We respectfully disagree with the concern on perplexity and performance trends. Our analysis shows that minor perplexity differences have little impact on model results. The primary goal was to show that LOGO maintains its language modeling ability, and performance should be assessed using benchmarks like LongBench, not just perplexity.\n\n---\n\n**[Question]** Lack of breakdown of evaluation details\n\n**[Re]** To provide clarity, we first present the partial context length distribution of LongBench. We can find that LongBench has a tightly concentrated length distribution, making it difficult to observe clear performance improvements across different length ranges.\n\n| Length Distribution         | 0-8K (%) | 8K-16K (%) | 16K+ (%) |\n|--------------------------------------|----------|------------|----------|\n| S-Doc QA                  | 66.0%   | 13.1%     | 20.9%   |\n| M-Doc QA    | 39.3%   | 60.3%     | 0.3%    |\n| Summ     | 62.5%   | 32.2%     | 5.3%    |\n\nWe encourage the reviewer to refer to our additional experiments on LongBench V2 `(Reviewer icM3 Concern 2)` and Ruler. These benchmarks feature a broader and more diverse context length distribution. The results demonstrate that LOGO consistently improves the backbone model’s performance across all context lengths, with particularly significant gains at longer context lengths (128K), further validating LOGO's effectiveness for long context.\n\n---\n\nWe hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.", "claims_and_evidence": "Significant improvement on LongBench: This claim is mostly valid. However, some comparisons could be more rigorous. For example, in Table 1, YaRN is compared with LOGO, but the version of YaRN used is training-free. A fairer comparison would include a trained version of YaRN, as prior work has shown that training with context extrapolation methods (like YaRN) improves performance.\nPerformance improvement on synthetic tasks (e.g., \"needles\" evaluations): Well-supported by experiments in Figure 3 and Section 4.2.\nNo degradation on short-context evaluations and reduced hallucinations: Clearly supported by results in Section 4.3.", "methods_and_evaluation_criteria": "The authors use LongBench to evaluate the model. While LongBench is a reasonable choice, other benchmarks like Ruler would also be valuable.\nLongBench and LongBench-v2 truncate input sequences from the middle, which may lead to different behaviors compared to evaluations with full-context retention. For example, GPT-4o excels on LongBench-v2 but lacks retrieval strength on Ruler. Including evaluations that retain full-context length would make the claims more robust.", "theoretical_claims": "Briefly went through the bound analysis in Appendix C and it seems to be valid but did not look into detailsBriefly reviewed the bound analysis in Appendix C—it appears valid, but I did not examine it in depth.", "experimental_designs_or_analyses": "More details on the training process of competitor models would help clarify the fairness of comparisons. For instance:\nLLaMA-2-7B-Chat-4K is compared against LOGO trained with Data-Engineering-80K, but the datasets have different token counts (5B tokens for Data-Engineering-80K vs. LOGO).\nKey hyperparameters like learning rate, batch size, and their impact on LOGO's number of negative samples are not discussed.\nSFT Regularization Robustness Claim (Section 5.1):\nThe paper states that LOGO is robust to the SFT regularization term, as perplexity drops while task performance remains stable.\nHowever, perplexity may not be a reliable indicator of long-context performance, as noted in previous research.\nAdditionally, it is unclear where the reported perplexity values come from (e.g., from the validation dataset during training or an evaluation set).\nThe inverse trend between LongBench scores and perplexity suggests that more evaluations are needed to fully validate the model's improvement.", "relation_to_broader_scientific_literature": "The paper has adequate coverage of prior methods for long-context training at the SFT stage.\nHowever, literature on preference training for LCMs is scarce, making direct comparisons with LOGO difficult.\nThe authors could reference additional related work on topics like memory compression and attention sparsity.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The strengths:\nNovel idea on constructing long-context preference data synthetically. Prior literature on this is relatively scarce.\nConducted adequate experiments with various prior methods including Data Engineering and Extrapolation methods.\nResults on long and short context tasks and analysis on the effectiveness of the proposed method.\nDetails on implementations on modeling and framework level make reproduction easier for audience.\nThe weaknesses:\nSome comparisons may not be fair enough. Ex. comparing LOGO with training free YARN.\nEvaluations on more benchmarks like Ruler will be helpful in providing a full picture of the method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can we see a breakdown of performance across different context lengths in LongBench?It would be insightful to observe how performance varies with context length. This would help clarify whether gains come from data quality rather than the proposed method itself. A visualization of scores over context length could provide better insights into its impact.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C3YoSBkPPb", "reviewer": "Reviewer_6VnN", "summary": "The paper introduces LOGO, a novel and efficient preference optimization strategy designed for long-context alignment in large language models (LLMs). LOGO addresses issues of misaligned responses in long-context models (LCMs) by introducing:\n- A Reference-Free Preference Optimization Strategy.\n- Efficient Data Synthesis for Long-Context Preference Optimization.\n- Positional Indices Synthesis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6VnN, thanks for your insightful comments and suggestions. Below is our detailed response.\n\n---\n\n**[Question 1]** Reasoning Capability Evaluation\n\n**[Re]** Thanks for raising this important point. However, our primary objective is **not to enhance reasoning per se**, but rather to **mitigate misalignment issues in long-context responses**.\n\nNevertheless, we have extended our experiments with LongBench V2 in this rebuttal, which is specifically designed for real-world long-context reasoning (`Reviewer icM3’s Concern 2`). Our results indicate **significant performance improvements even in reasoning tasks**, particularly in settings ''without chain-of-thought (CoT) prompting''. Notably, Llama3-8B-80K achieves an 8-point average improvement, and Qwen2.5-7B-Instruct  yields a 2.6 average point improvement, suggesting that LOGO training may **implicitly enhance reasoning capabilities**—potentially due to the presence of multi-hop data in our synthesis preference data.  \n\nTo achieve reasoning performance on par with OpenAI’s O1 or DeepSeek’s R1, stronger RL supervision signals, such as *Long-CoT*, would likely be required. Interestingly, we have recently been investigating this direction: by enabling LCM to actively search for and integrate key information from long-context inputs before generating responses, we observe substantial performance gains. However, this exploration falls beyond the scope of LOGO itself and is an avenue for future work.  \n\n---\n\n**[Question 2]** Efficiency Gains vs. Traditional RLHF Approaches\n\n**[Re]**\nA key challenge in applying traditional RLHF methods (e.g., PPO, DPO) to long-context modeling (LCM) is their heavy reliance on reward models, critical models, etc. For instance, PPO requires a reward model, a reference model, and a value model, while DPO depends on a reference model. \n\nHowever, in the long-context domain, there are no publicly available reward models or value models to facilitate such training.  To address this, LOGO introduces an alternative: importance scoring and a novel training objective to replace the need for an offline reward model. This significantly improves training efficiency and eliminates the dependency on additional models.  \n\nBesides, among open-source works, we have found only one study similar to ours: LongPO, which follows a traditional DPO-based approach and requires an additional reference model for training. To provide a clearer comparison, we present the following table highlighting the key differences between SFT, LOGO and LongPO: \n\n| Training Strategy          | Memory/GPU | Bsz/GPU | Total Throughput (8 GPUs) | Actual Training Length | Training Time (2,000 steps) | Real-world Task | Language Modeling |\n|----------------------------|------------|---------|---------------------------|------------------------|-----------------------------|-----------------|-------------------|\n| SFT                        | 80GB       | 1       | 8 samples                 | 64K                    | 14h                         | 43.2            | 6.6               |\n| SFT + Ring Attention       | 45GB       | 1       | 4 samples*                | 128K                   | 24h                         | 44.3            | 6.6               |\n| LOGO (w/o ref model) + Ring Attention      | 69GB       | 3       | 12 samples*               | 64K                    | 30h                         | 47.7            | 9.8               |\n| LOGO (w/o ref model) | 64GB       | 3       | 24 samples                | 12K                    | 16h                         | 47.0            | 10.4              |\n| LongPO (**w/ ref model**)  | OOM        | 2       | 16 samples                | 64K                    | -                         | -            | -              |\n| LongPO (**w/ ref model**) + Ring Attention (ring_size=2) | 62GB  | 2       | 8 samples  | 64K                    | >24h             | 44.7            | 17.6              |\n\nWe observed that without employing the **Ring Attention** strategy or a CP-parallel approach, LongPO cannot be deployed on one 80GB GPU due to the combination of long input sequences and the requirement for an additional reference model. Even with the Ring Attention strategy—where every 2 GPUs process a segment of the sequence in parallel, the training time for 2,000 steps exceeds 24 hours. Additionally, LongPO's performance is inferior to LOGO, as it only applies a simplistic preference data processing method, where shorter responses are treated as preferred replies.\n\nIn short, even in settings with **ample computational resources**, LOGO may still remain beneficial due to its **scalability and reduced model dependency**, making it a more practical and efficient alternative to traditional RLHF methods for long-context alignment.  \n\n---\n\nWe hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria proposed in this paper appear well-justified and appropriate for the long-context alignment", "theoretical_claims": "This paper does not contain significant theoretical proofs to verify.", "experimental_designs_or_analyses": "The authors have conducted extensive experiments to validate the effectiveness of LOGO. The results demonstrate that LOGO significantly improves long-context alignment while maintaining efficiency.", "relation_to_broader_scientific_literature": "This paper makes significant contributions to the broader scientific literature on long-context alignment in LLMs. Prior works have primarily focused on scaling context window sizes (e.g., through post-training on long-instruction data, novel architectures, or positional encoding modifications). However, research has shown that large-context windows alone do not guarantee alignment, as models still exhibit hallucinations and fail to follow instructions effectively.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- The writing is well-structured, with clear logic and readability, making the paper easy to follow.\n- The proposed LOGO method is a novel and efficient approach to long-context alignment, demonstrating both effectiveness and scalability.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- Applicability to Reasoning Tasks:\nLOGO focuses on long-context alignment and preference optimization. However, recent advances, such as OpenAI's O1 and DeepSeek's R1, have shown that test-time scaling can significantly enhance reasoning ability. Have you evaluated whether LOGO can also improve the model’s reasoning capabilities, particularly in long-context reasoning tasks?\n- Efficiency Gains vs. Traditional RLHF Approaches\nHow does LOGO compare against standard RLHF approaches like PPO in terms of GPU usage and training time? Would LOGO still be beneficial in a setting with ample computational resources?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces LOGO, a novel preference optimization method tailored for long-context alignment in LLMs. The method is efficient, reference-free, and shows consistent performance improvements across multiple long-context benchmarks with a modest training budget. While some aspects are incremental, the authors provide thorough empirical validation and address reviewer concerns well in the rebuttal. I lean toward recommending acceptance based on its practical contributions to long-context alignment.
Decision: Accept (poster)