LOGO --- Long cOntext aliGnment via efficient preference Optimization

## Abstract

Abstract Long-context models (LCMs) have shown great potential in processing long input sequences (even more than 100M tokens) conveniently and effectively.
With significant progress, recent research has pointed out that LCMs can accurately locate token-level salient information within the context.
Yet, the generation performance of these LCMs is far from satisfactory and might result in misaligned responses, such as hallucinations.
To enhance the generation capability of LCMs, existing works have investigated the effects of data size and quality for both pre-training and instruction tuning.
Though achieving meaningful improvement, previous methods fall short in either effectiveness or efficiency.
In this paper, we introduce LOGO (Long cOntext aliGnment via efficient preference Optimization), a training strategy that first introduces preference optimization for long-context alignment.
To overcome the GPU memory-bound issue caused by the long sequence, LOGO employs a reference-free preference optimization strategy and adopts a position synthesis method to construct the training data.
By training with only 0.3B data on a single 8 × \times × A800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K model to achieve comparable performance with GPT-4 in real-world long-context tasks while preserving the model’s original capabilities on other tasks, e.g., language modeling and MMLU.
Moreover, LOGO can extend the model’s context window size while enhancing its generation performance.

## 1 Introduction

With the rapid advancements of Large Language Models (LLMs), handling long contexts (even more than 100M tokens (anthropic, )) has become a fundamental capability for recent LLMs.
This further unlocks the potential of LLMs for novel tasks and applications, e.g., code analysis (Zhu et al., ), while simultaneously eliminating the need for complex toolchains and intricate workflows that were previously required to overcome the context-length constraints (Ravaut et al., ).

Yet, recent studies have pointed out that these long-context models (LCMs) failed to achieve satisfactory performance in long-context tasks, where LCMs might produce misaligned results, such as instruction unfollowing and hallucinations (Belyi et al., ; Zhang et al., ).
To mitigate the above issue, the open-source community has made significant efforts, primarily focusing on building high-quality long instruction data and extending the data size (Wu et al., ; Bai et al., ; Fu et al., ; Bai et al., ).
As shown in Fig. 1, though achieving meaningful improvement, these methods fall short in effectiveness or efficiency.
For instance, the Llama-3.1-8B-128K model AI@Meta () was pre-trained on around 300B long instruction data, but it even underperforms the Llama-3-8B-Instruct-80K model (Zhang et al., ), which was post-trained with 1.5B high-quality long instruction data based on the Llama-3-8B-Instruct model (AI@Meta, ).
As for the Llama-3-8B-Instruct-80K model, it shows slight improvement compared to the baseline and still lags greatly behind the closed-source counterparts like GPT-4 (Achiam et al., ).

Recently, Wu et al. () pointed out that LCMs can accurately locate token-level salient information within the context.
As shown in Fig. 1(b), we visualize the information retrieval capability(^1^11Retrieval capability is reflected through the recall score of salient tokens located by retrieval heads (Wu et al., ). We calculate the average recall score across the top-10 retrieval heads. A higher retrieval score indicates that the LCM can retrieve more critical information. Details are shown in Appendix B.) (reflected by the retrieval score) and the generation capability (reflected by the recall score) of different LCMs on the synthetic retrieval task, where we can observe a minimal difference among the retrieval scores from various LCMs, but large differences in their generation performance.
This suggests that while LCMs are adept at identifying key information within long contexts, they struggle to effectively utilize the retrieval information for generation.
The underlying cause might be the commonly used training approach of LCMs, which relies on token-level maximum likelihood loss, i.e., Cross-Entropy (CE) loss, calculated on both the context and the predictions.
Given that the context’s sequence length is typically much longer than the prediction portion, the feedback signal (CE loss) from the prediction is often overshadowed by that from the context.
As a result, the CE loss becomes ineffective in optimizing the generation capabilities of LCMs.

To effectively optimize LCMs for generating desired outputs and avoid misaligned results, this paper introduces LOGO (Long cOntext aliGnment via efficient preference Optimization), the first training strategy that incorporates preference optimization for long-context alignment.
There are two key components in LOGO: (1) a training objective designed to guide LCMs to distinguish between preference predictions (i.e., correct outputs) and dis-preference predictions (e.g., misaligned outputs like hallucinations), and (2) a corresponding data construction pipeline that only involves open-source models.
It is worth noting that training with long sequence data is a memory-intensive task (Dao, ) and the DPO algorithm also has a high GPU memory demand.
To overcome the GPU memory-bound and improve the training efficiency, LOGO adopts a reference-free training objective and the positional indices synthesis method (Zhu et al., ).
Consequently, we can perform the LOGO training with only 0.3B data on a single 8$\times$A800 GPU machine within 16 hours.

By training with LOGO, LCMs can achieve significant improvements in real-world tasks and gain moderate improvements in synthetic and language modeling tasks, as well as maintaining good performance on the short-context tasks, e.g., MMLU (Hendrycks et al., ).
As shown in Figure 1(a), our Llama-3-8B-LOGO significantly outperforms GPT3.5-Turbo in real-world tasks and approaches the performance of some top closed-source models like GPT-4.
Additionally, LOGO can also generalize to the training of short-context LLMs such as Llama-2-7B-Chat-4K (Touvron et al., ), which can potentially extend their context window size up to 8 times (e.g.,32K context window size for Llama-2-7B-Chat-4K) while simultaneously enhancing their performance substantially.

## 2 Related Work

### 2.1 Long Context Scaling and Long Context Alignment

Two steps are essential for empowering LLMs with the ability to handle long-context tasks: 1) context scaling, which expands the limited context window size to support long-context tasks, e.g., from 8k to 128k; and 2) long-context alignment, which ensures that LCMs can follow long instructions.
Currently, the open-source community mainly focuses on the former, primarily by (1) post-training models on long instruction data (Chen et al., ; Xiong et al., ; Fu et al., ; Zhang et al., ), (2) devising novel model architectures (Yang et al., ; Zhang, ; Tworkowski et al., ), and (3) modifying positional encoding (Peng et al., ; Chen et al., ; Jin et al., ) to extend the context window of LLMs.
However, current works (Belyi et al., ; Hsieh et al., ; Zhang et al., ) indicated that LCMs still underperform in long-context tasks, frequently manifesting issues such as hallucinations and failure to follow instructions, despite possessing large context window size.
To mitigate this issue, Bai et al. () and Wu et al. () proposed to align the LCMs in long-context scenarios by synthesizing long-dependency instruction data to fine-tune the models.
Some LLMs are even pre-trained with massive long instruction data (Jiang et al., ; Dubey et al., ; Abdin et al., ).
Yet, despite numerous attempts that have been made to improve the data quality and quantity, the performance of open-source LCMs still lies far behind close-source LCMs.
Therefore, focusing solely on data augmentation methods can not resolve the long-context alignment problem efficiently and effectively.
In this work, we address the above issue from the training objective perspective.
Building upon the language modeling task, we introduce LOGO, which contains a long-context preference optimization training objective.
Experimental results demonstrate that, with a small amount of data and computational resources, LOGO can significantly enhance the generation capability of LCMs.

### 2.2 Model Alignment with Direct Preference Optimization

Direct Preference Optimization (DPO) (Rafailov et al., ) is a widely adopted RLHF algorithm (Ouyang et al., ) that aims to align models with human preferences.
Compared to other reinforcement learning methods, e.g., PPO (Schulman et al., ), DPO can achieve strong performance while eliminating the need for a separate reward model.
Unlike Supervised Fine-Tuning (SFT), which guides LLMs to fit predictions to ground truth at the token level, DPO updates the model parameters with discrete evaluation scores.
Specifically, DPO teaches the model to “reject” misaligned responses and “accept” preferred responses with differently assigned prediction scores.
Significant efforts have been made to enhance the effectiveness and efficiency of DPO, such as CPO (Xu et al., ), TPO (Saeidi et al., ), and ORPO (Hong et al., ).
Among them, SimPO (Meng et al., ) utilizes the average log probability of a sequence as the implicit reward, which better aligns with the generation tasks and eliminates the need for a reference model.

## 3 Methodology

### 3.1 Background

##### Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO)

DPO is one of the most popular offline preference optimization strategies in RLHF (Rafailov et al., ).
Given prompt $x$, DPO aims to maximize the likelihood of a preferred response $y_{w}$ over a dis-preferred one $y_{l}$, thereby preventing the model from generating undesired content.
There are three essential modules in the DPO training process: one reference model and one policy model for calculating the DPO loss jointly, and one evaluation strategy (or evaluation model) for distinguishing between $y_{w}$ and $y_{l}$.
SimPO (Meng et al., ) is an improved variant of DPO, which employs an implicit reward formulation that directly aligns with the generation metric, e.g., PPL, thereby eliminating the need for a reference model.
The training objective of SimPO can be written as:

$$ $\mathcal{L}_{\mathrm{SimPO}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w},y_{l})}\left[ \log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-\frac{\beta}{|y _{l}|}\log\pi_{\theta}(y_{l}|x)-\gamma\right)\right],$ (1) $$

where $\pi_{\theta}$ is the policy model (model to be optimized), $\beta$ (scaling of the reward difference) and $\gamma$ (target reward margin) are the hyper-parameters to separate the preferred and dis-preferred responses.

##### Efficient Context Scaling with Positional Indices Synthesis

Transformer-based models rely on positional indices to identify the relative position of each token (Raffel et al., ).
One efficient method to extend the data context length is modifying the positional indices to simulate long-sequence inputs without altering the real input sequence (Press et al., ; Ruoss et al., ).
By default, the positional indices of a sequence of length $k$ are $\mathcal{P}(k)=\{0,1,\cdots,k-1\}$.
To extend the sequence length from $k$ to $K$, we can synthesize the positional indices: $\mathcal{P}_{\mathcal{B}}(K)=\{0+b_{0},1+b_{1},\cdots,k-1+b_{k-1}\}$, where $\mathcal{B}=\{b_{0},b_{1},\cdots,b_{k-1}\}$ is the positional bias applied to each original position index and $k-1+b_{k-1}=K$.
To ensure effectiveness, the synthesis of position indices should achieve a uniform distribution of relative distances within the extended sequence length $[0,K]$ and cover as many of the extended positional indices as possible (Wu et al., ).

### 3.2 Long-context alignment with LOGO

#### 3.2.1 Training Objective of LOGO

In long-context scenarios, LCMs are prone to generating various misaligned responses, such as hallucinations and failing to follow instructions (Belyi et al., ).
However, there is a lack of effective strategies (or models) to detect these misaligned outputs, posting a great challenge for selecting preference and dis-preference samples in preference optimization (we will elucidate this in Appendix C, where we also show the misalignment cases).
Therefore, instead of finding one dis-preference instance with a specific error pattern, we can expand the dis-preference space to push the model away from a range of possible dis-preference instances.
We design the loss function based on SimPO (Eq. 1), as it is more aligned with the generation tasks and free of the reference model, which is efficient for long-context training.
The training objective can be written as:

$$ $\mathcal{L}_{\mathrm{LOGO}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w},y_{l}^{(1 \cdots M)})}\left[\log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}| x)-\frac{\beta}{M|y_{l}|}\sum_{j=1}^{M}\log\pi_{\theta}(y_{l}^{(j)}|x)-\gamma \right)\right],$ (2) $$

where $M$ is the number of dis-preference instances.

Furthermore, to avoid reward hacking phenomenon (Yuan et al., ; Hong et al., ) as well as preserve the modeling capabilities of LCMs, we add an SFT regularization term in Equ 2.
This regularization term serves to prevent the policy model $\pi_{\theta}$ from drifting away from its original capabilities acquired through SFT.
The final loss function of LOGO can be written as:

$$ $\mathcal{L}_{\mathrm{LOGO}}^{*}(\pi_{\theta})=\mathcal{L}_{\mathrm{LOGO}}(\pi_ {\theta})+\lambda\mathbb{E}_{(x,y_{w})}\log\pi_{\theta}(y_{w}|x)),$ (3) $$

where $\lambda$ is the hyper-parameter that controls SFT regularization term.

#### 3.2.2 Training Dataset Construction of LOGO

To perform the LOGO training, we introduce a tailored LOGO dataset construction pipeline.
For each long-context sample, we can format it as a triplet $\mathcal{X}=\{Q,\mathcal{C},P\}$, where $Q$, $\mathcal{C}$, and $P$ represent the question, reference context, and the model prediction, respectively.
As shown in Fig. 2, to construct training data for LOGO, we first divide the context $\mathcal{C}$ into equal-length chunks $\{C_{1},C_{2},\cdots,C_{n}\}$.
Then, three steps are involved: (1) Importance Scoring with Automatic Evaluator, (2) Preference and Dis-preference Data Synthesis, and (3) Positional Indices Synthesis.

Figure: Figure 2: Dataset construction pipeline of LOGO.
Refer to caption: x2.png

##### Importance Scoring with Automatic Evaluator

To construct preference (aligned) and dis-preference (misaligned) data in long-context scenarios, an efficient method is to guide the model to respond based on different contexts.
Specifically, to construct the preference data, we only provide the model with context relevant to the question, thus enhancing the fidelity of the model’s output by reducing contextual interference (Shi et al., ).
Conversely, we can add more irrelevant context to guide the model in generating misaligned content like hallucinations.
To find the relevant chunks $C_{i}$ within the context, we utilize an automatic evaluator $\mathrm{Eval}(\cdot)$ to calculate the “contribution” of each chunk $C_{i}$ to the question $Q$.
Specifically, we utilize an $\mathrm{Eval}(\cdot)$ to identify all the entities within a chunk $C_{i}$.
The more overlapping entities $C_{i}$ shares with the question $Q$, the greater its influence on the final prediction, allowing us to assign a higher score to this chunk.
With $\mathrm{Eval}(\cdot)$, we efficiently assign importance scores $S=\{s_{1},s_{2},\cdots,s_{n}\}$ to all the chunks.

##### Preference and Dis-preference Data Synthesis

To construct preference and dis-preference data based on the model prediction $P$, we select and combine the chunks mentioned above to create diverse contexts, guiding the model to generate different outputs.
Let $N$ represent the number of chunks within a context, and we define a threshold $\delta$ to distinguish between critical and irreverent chunks.
Specifically, chunks $\mathcal{C}_{>\delta}$ scoring above $\delta$ are considered as essential chunks while chunks $\mathcal{C}_{<\delta}$ scoring below $\delta$ are considered as irreverent chunks.
Then, we combine $Q$ and $\mathcal{C}_{>\delta}$ for model to generate preference prediction $P_{\mathrm{preference}}$, and adjust the ratio of chunks sampled from $\mathcal{C}_{>\delta}$ and $\mathcal{C}_{<\delta}$ for model to generate dis-preference predictions $P_{\mathrm{dis-preference}}$.
Specifically, $P_{\mathrm{dis-preference}}$ is mainly sampled from two misaligned error patterns: (1) model generation based on all irrelevant chunks $P_{\mathrm{dis-preference}}^{\prime}$, and (2) model generation based on partially relevant chunks $P_{\mathrm{dis-preference}}^{\prime\prime}$.
The above data construction process can be written as:

$$ $\left\{\begin{aligned} &P_{\mathrm{preference}}=\pi_{\theta}(Q,\mathcal{C}_{> \delta})~{}\mathrm{,where}~{}~{}\mathcal{C}_{>\delta}\sim\mathcal{C},|\mathcal {C}_{>\delta}|=N\\ &P_{\mathrm{dis-preference}}\sim\left\{\begin{aligned} &P_{\mathrm{dis- preference}}^{\prime}=\pi_{\theta}(Q,\mathcal{C}_{<\delta})~{}\mathrm{,where}~ {}~{}\mathcal{C}_{<\delta}\sim\mathcal{C},|\mathcal{C}_{<\delta}|=N~{},\\ &P_{\mathrm{dis-preference}}^{\prime\prime}=\pi_{\theta}(Q,\mathcal{C}_{< \delta},\mathcal{C}_{>\delta})~{}\mathrm{,where}~{}~{}\mathcal{C}_{<\delta}, \mathcal{C}_{>\delta}\sim\mathcal{C},|\mathcal{C}_{<\delta}\cup\mathcal{C}_{> \delta}|=N\end{aligned}\right\}\end{aligned}\right..$ $$

Subsequently, the constructed preference and dis-preference data share the same context $\mathcal{C}^{\prime}$, which is combined with all the chunks in $\mathcal{C}_{>\delta}$ and partial chunks in $\mathcal{C}_{<\delta}$. Finally, one LOGO training sample can be written as $\left(\{Q,\mathcal{C}^{\prime},T_{\mathrm{preference}}\},\{Q,\mathcal{C}^{
\prime},T^{(i)}_{\mathrm{dis-preference}}\}_{i=1}^{M}\right)$, which is consistent with Eq. 3.

##### Positional Indices Synthesis

Given that each LOGO training sample includes $(M+1)$ instances, with one preference instance and $M$ dis-preference instance, a long context length of $\mathcal{C}^{\prime}$ can easily lead to GPU memory overflow (even on GPUs with 80GB memory).
To address this, we employ a positional encoding synthesis strategy.
By assigning different synthetic positional indices to each chunk, we can simulate long-sequence training data with short context data (Wu et al., ).
Specifically, to ensure that the synthetic positional indices do not disrupt the semantic structure of short context, the positional indices within each chunk should be continuous, while indices between adjacent chunks can be discrete, i.e., omitting certain positional indices (as shown in sub-Fig. ③ in Fig. 2).
Given $N$ equal-length chunks within each sample(^2^22Since the length of question $\mathcal{Q}$ and prediction $P$ are much shorter compared to the long context $\mathcal{C}$, we can ignore the length of $\mathcal{Q}$ and $\mathcal{P}$ for simplicity.), to achieve a uniform distribution of relative distance within the expanded context length $[0,K]$, each positional bias term $b_{i}\in\mathcal{B}$ should be sampled from a uniform distribution.
The synthetic positional indices can be written as:

$$ $\mathcal{P}_{\mathcal{B}}(K)=\{i+b_{i}\}_{i=0}^{k-1},~{}\mathrm{where}~{}~{}b_ {i}\sim\mathcal{U}(1,(i\bmod|C_{i}|)\times(K-k)/N),$ (4) $$

where $\left(i\bmod|C_{i}|\right)$ indicates the chunk index where the current positional index $i$ resides, and $(K-k)/N$ represents the expansion size for each chunk. More details are shown in Appendix D.

## 4 Experiment

### 4.1 Settings

##### LOGO Dataset Construction

We construct the LOGO datasets based on two corpora: (1) 4,000 instances sampled from long-llm-data(^3^33[https://huggingface.co/datasets/namespace-Pt/long-llm-data](https://huggingface.co/datasets/namespace-Pt/long-llm-data)) (Zhang et al., ), which includes reference contexts from multiple domains (e.g., biography, paper, etc.) and questions generated by GPT-4, covering tasks such as Single-Detail QA, Multi-Detail QA, and Summarization; (2) 2,000 instances sampled from RedPajama (Computer, ) to mitigate forgetting, where we prompt the open-source LCM Qwen2-70B-Instruct (Yang et al., ) to generate questions for each instance.
Then, we split each instance into equal-length chunks, with each chunk containing 512 tokens.
To construct preference and dis-preference data, we use the spaCy model(^4^44[https://spacy.io/usage/models](https://spacy.io/usage/models)), a named entity recognition (NER) model that can identify all the entities within a context, as the evaluator $\mathrm{Eval(\cdot)}$.
We use the number of overlapping entities between each chunk $C_{i}$ and the question $Q$ as the importance score.
We set the threshold $\delta$ as 6, and chunk number $N$ as 16, i.e., selecting and combining 16 chunks as the reference context for training.
As for the number of dis-preference instances in the LOGO training objective, we set $M=2$, i.e., each training sample includes one preference instance and two dis-preference instances.
Then, we apply Eq. 4 to construct positional indices for each instance within each sample. Specifically, we adopt two different sampling strategies on positional bias $\mathcal{B}$ to ensure that all positional indices are uniformly covered and maintain the semantic structure of the context (see Appendix D for more details).
After positional indices synthesis, we have a total number of 12,000 training samples, with a total data size of approximately 12,000$\times$512$\times$16$\times$3$\approx$0.3B tokens.

##### Training Settings

To improve the training efficiency while preserving the inherent capabilities of the LLMs, we freeze the backbone model and apply LoRA (Hu et al., ) method, which only fine-tunes the attention and token embedding modules, to perform training.
Additionally, thanks to positional indices synthesis, LOGO can potentially scale the context length and ensure alignment in long-context tasks simultaneously.
Therefore, we experiment with two type of models: (1) Short-context Models (SCMs) including Llama-2-7B-Chat (Touvron et al., ) and Llama-3-8B-Instruct (AI@Meta, ), which own context lengths of 4K and 8K, respectively; and (2) Long-context Models (LCMs), including Llama3-8B-Instruct-80K (Zhang et al., ), Llama-2-7B-Instruct-80K (Fu et al., ) and Mistral-Instruct-7B-V0.2 (Jiang et al., ), which inherently have long context windows.
For SCMs, given that excessive scaling with positional indices synthesis method can result in the missing of some positional indices, potentially impacting model performance, we scale the context windows of SCMs to 8 times of their original context length.
For LCMs, we maintain their original context length.
To accelerate the training process and save GPU memory, we adopt DeepSpeed Zero 3 (Aminabadi et al., ).
All the experiments are conducted on a 8$\times$A800 (80GB) GPU machine, and the training is completed within 16 hours.
For the setting of hyper-parameters $\beta$ and $\gamma$ in Eq. 2, we adhere to the recommendations provided in Meng et al. () for different models, where $\beta=10,\gamma=3$ for Llama-3-8B-based model, $\beta=2.5,\gamma=0.25$ for Mistral-Instruct-7B-V0.2-based model, and $\beta=3,\gamma=0.6$ for Llama-2-7B-based model.
We set $\lambda=0.1$ in Eq. 3 for SFT regularization to stabilize the training process of LOGO and prevent the reward hacking phenomenon mentioned above.

##### Evaluation Settings

We assess the LOGO training strategy across three categories of long-context tasks: real-world long-context tasks, a synthetic retrieval task, and the language modeling task.
To explore the impact of LOGO training in short-context scenarios, we also evaluate models on short-context tasks.
For comparison, we select two representative context scaling methods: YaRN (Peng et al., ) and RandPOS (Ruoss et al., ), as well as two types of long-instruction tuning strategies Xiong et al. (), i.e., calculating loss on the entire sequence (Full) and the prediction (Partial).
We select LongAlpaca (Chen et al., ) corpus as the instruction training data, which contains 12,000 long instruction samples with each sample containing 32K context length.

### 4.2 Performance on Long-context Tasks

##### Results on Real-world Long-context Tasks

**Table 1: Evaluation results on LongBench benchmark, where $\dagger$ denotes training-free method.**
| Models | S-Doc QA | M-Doc QA | Summ | Few-shot | Synthetic | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-3.5-Turbo-16K | 39.8 | 38.7 | 26.5 | 67.1 | 37.8 | 42.0 |
| LongChat-v1.5-7B-32k | 28.7 | 20.6 | 26.7 | 60.0 | 15.8 | 30.4 |
| LLama-3.1-8B-Instruct-128K | 23.9 | 15.8 | 28.9 | 69.8 | 57.5 | 39.2 |
| Results on SCMs (scaling $\times$8 context window) |  |  |  |  |  |  |
| Llama-3-8B-Instruct-8K | 39.3 | 36.2 | 24.8 | 63.5 | 39.9 | 40.7 |
| +  YaRN-64K^† | 38.0 | 36.6 | 27.4 | 61.7 | 40.9 | 40.9 |
| +  RandPOS-64K | 32.5 | 30.5 | 26.5 | 61.3 | 33.4 | 36.8 |
| +  LOGO-64K | 39.8 | 36.7 | 28.8 | 65.4 | 49.0 | 43.9 |
| Llama-2-7B-Chat-4K | 24.9 | 22.6 | 24.7 | 60.0 | 5.9 | 27.6 |
| +  LOGO-32K | 26.7 | 23.3 | 26.3 | 63.1 | 11.1 | 30.1 |
| Results on LCMs (long-context alignment) |  |  |  |  |  |  |
| Llama-3-8B-Instruct-80K | 43.0 | 39.8 | 22.2 | 64.3 | 46.3 | 42.3 |
| +  Instruct Tuning (Full) | 38.8 | 35.0 | 24.6 | 65.9 | 44.5 | 41.8 |
| +  Instruct Tuning (Partial) | 39.3 | 36.2 | 26.8 | 63.5 | 48.0 | 42.8 |
| +  LOGO-80K | 44.0 | 41.2 | 28.1 | 68.6 | 53.0 | 47.0 |
| Llama-2-7B-Instruct-80K | 26.9 | 23.8 | 21.3 | 65.0 | 7.9 | 29.0 |
| +  LOGO-80K | 33.6 | 28.0 | 29.4 | 65.1 | 24.5 | 36.1 |
| Mistral-Instruct-7B-V0.2-32K | 31.7 | 30.6 | 16.7 | 58.4 | 17.9 | 31.1 |
| +  LOGO-32K | 38.3 | 37.6 | 26.1 | 67.0 | 31.5 | 40.1 |

We evaluate the LOGO performance with real-world long-context tasks in LongBench (Bai et al., ), a comprehensive benchmark suite encompassing 16 distinct datasets spread across 6 task categories, including Single Document QA (S-Doc QA), Multi-Document QA (M-Doc QA), Summarization (Summ), Few-shot, Synthetic, and Code.
It is worth noting that we exclude the Code category since the code testing data primarily involves contexts of just around 4,000 tokens and our training data does not cover this domain.
We report the evaluation results in Tab. 1, where we can observe that: (1) LOGO achieves the best performance among all the settings. Specifically, for SCMs, LOGO outperforms both YaRN and RandPOS. Although these two methods can potentially extend the context window of SCMs, they significantly impair performance on real-world long-context tasks. For instance, RandPOS causes the Llama3-8B-Instruct model to drop around 6 points on average compared to the baseline, with particularly notable declines in performance on the synthetic tasks.
For LCMs, LOGO can significantly improve model performance, with all LCMs showing varying degrees of improvement, e.g., Llama-3-8B-Instruct-80K model shows an average 5-point improvement compared to the baseline, whereas the instruct tuning method tends to restrict even a well-performing LLMs to a limited performance bottleneck;
(2) Compared to other methods, LOGO demonstrates significant improvement in information-intensive tasks, which require the model to gather information from various parts of the context.
Specifically, in summarization and synthetic tasks, LCMs trained with LOGO can achieve significant performance improvements, with at least a 5-point increase.

##### Evaluation Results on Synthetic Retrieval Task

Figure: Figure 3: Results of the Needle-in-a-Haystack testing.
Refer to caption: x3.png

To investigate whether the LOGO training strategy affects the information retrieval capabilities of LCMs, we conduct a Needle-in-a-Haystack testing (gkamradt, ).
More concretely, NIAH is a synthetic retrieval task that evaluates a model’s ability to retrieve key information (needle) from any position within its context window.
We employ a color scale ranging from light green (indicating a 100% successful recall), to red (indicating a complete failure).
Our test covers context lengths from 8K to 88K, with intervals of 0.5K and the needle at various depths.
As shown in Fig. 3, we can find that LOGO can scale the context window for SCMs (left group) and does not adversely affect the original context window size of LCMs (right group).
We can also observe that the original LCMs (middle group) and those trained with LOGO (right group) share similar patterns, i.e., similar shades of color, yet LOGO improves performance in areas where the original LCMs fail.
This indicates that LOGO does not compromise the inherent capabilities of LCMs but rather enhances their original weakness.

Figure: Figure 4: Evaluation results of language modeling task. The solid and dashed curves represent the PPL of the baselines and LOGO, respectively.
Refer to caption: x4.png

We can also find that the Llama-3-8B-8K model demonstrates superior context scaling effects compared to the Llama-2-7B-4K model. This can be attributed to the larger RoPE base value in Llama-3-8B-8K (500,000) compared to Llama-2-7B-4K (10,000), which has been proven to facilitate more effective scaling of the context window size (AI@Meta, ).

##### Evaluation Results on Language Modeling Task

We test the language modeling capability of LCMs by calculating the Perplexity (PPL) on the Gutenberg (PG-19) testing set (Rae et al., ), with context lengths ranging from 2K to 64K.
Considering that extremely long context lengths can cause the PPL calculation to exceed GPU memory, we apply the sliding window approach proposed by Press et al. ().
As depicted in Fig. 4, for LCMs, such as Llama-3-8B-Instruct-80K and Llama-2-7B-Instruct-80K, using LOGO does not compromise the language modeling capability since the solid line (PPL of the backbone model) and the dashed line (PPL of LOGO) almost completely overlap.
In the case of SCMs, such as the Llama-3-8B-Instruct-8K model, LOGO not only effectively scales the context window size of baseline models (the purple dotted curve versus the purple solid curve) but also achieves a lower PPL score compared to the SFT method since the yellow dotted curve (PPL of Llama-3-8B-Instruct-LOGO) is much lower than the blue solid curve (PPL of Llama-3-8B-Instruct-80K).

### 4.3 Performance on Short-context Tasks

To investigate whether LOGO training affects model performance on short-context tasks, we select three widely used benchmarks for assessing LLMs’ foundational capabilities that possess short input sequence: MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., ), and ARC (Hard and Easy) (Clark et al., ).
As illustrated in Fig. 5, we find that LOGO not only preserves the LLM’s inherent capabilities on short-context tasks but also demonstrates improvements in some specific tasks.
This is because LOGO aims to teach the model to generate responses based on the context rather than fabricating results (such as producing hallucinations), which is equally applicable to short-context tasks.
We can also find that scaling context length with LOGO yields better results than instruction tuning.
For instance, as demonstrated in the TruthfulQA task, Llama-3-8B-Instruct-80K shows significant performance degradation compared to the Llama-3-8B-Instruct-8K-LOGO (64K).
Such a phenomenon indicates a high “alignment tax” paid from instruction tuning (Fu et al., ).

Figure: Figure 5: Model performance on short-context tasks, including MMLU, TruthfulQA, and ARC.
Refer to caption: x5.png

## 5 Ablation Study

For ablation studies, we experiment with the Llama-3-8B-Instruct-80K model, which demonstrates strong baseline performance across the various tasks.
We conduct experiments on the real-world tasks by reporting the average score on LongBench (denoted with $\mathrm{LB}$), and the language modeling task by calculating the PPL score on the PG-19 testing set with a 64K context length.
In Sec. 5.1, we analyze the impact of different hyper-parameters in the LOGO training objective.
In Sec. 5.2, we discuss the impact of synthetic data of varying lengths.
In Sec. 5.3, we compare LOGO with SFT by visualizing LCM’s generation and information retrieval capabilities along the training phase.

Figure: Figure 6: Ablation study results. (a) Comparison among different settings on the language modeling task (PPL) and real-world tasks (Avg. score on LongBench testing set); (b) Reward difference distribution under different $M$ settings; (c) Training GPU memory consumption of different settings.
Refer to caption: x6.png

### 5.1 Analysis of LOGO Training Objective

##### Effect of SFT Regularization Term λ 𝜆 \lambda italic_λ

To investigate the SFT regularization term in Equ. 3, we adjust the value of $\lambda$ to control the SFT regularization term.
As depicted in Fig. 6(a), we can observe that increasing $\lambda$ enables the model to achieve a lower PPL score.
For real-world tasks, the impact of SFT regularization on the final results is minimal.
For example, for settings $(M=2,\lambda=0.1,Ctx.=8K)$, $(M=2,\lambda=0.5,Ctx.=8K)$, and $(M=2,\lambda=1.0,Ctx.=8K)$, we can observe that as $\lambda$ gradually increases, the PPL significantly decreases, with a difference of nearly 3.5 points, while the average score on LongBench only differs by around 1.5 points.

Figure: Figure 7: Comparison between SFT and LOGO training strategies on the synthetic retrieval task.
Refer to caption: x7.png

##### Effect of the Number of Dis-Preference Instances

We experiment with different numbers of dis-preference instance $M=\{1,2,3\}$ in Eq. 3. Specifically, when $M$ equals 1, the LOGO Objective degenerates into the SimPO Objective.
As shown in Fig. 6(a), using more dis-preference samples can enhance the model’s performance on real-world tasks, but it slightly impacts the capability for language modeling.
We also visualize the learned reward margin $r(x,y_{w})-\frac{1}{M}\sum_{i=1}^{M}r(x,y_{l}^{(i)})$ under various $M$ values in Fig. 6(b).
We can observe that using a larger $M$ can flatten the distribution and make it easier for the model to distinguish between preference and dis-preference samples as the gap between $r(x,y_{w})$ and $\frac{1}{M}\sum_{i=1}^{M}r(x,y_{l}^{(i)})$ gradually increases with larger $M$.
This is because increasing $M$ can cover more samples with various types of misalignment patterns.
However, as shown in Fig. 6(c), increasing $M$ poses a challenge as it may exceed GPU memory limits.
While introducing more dis-preference samples in the LOGO objective function might be beneficial, optimizing this in practical deployment is necessary.
Additionally, the impact of each dis-preference sample’s weight needs to be explored, which we will address in our further work.

### 5.2 Effect of Synthetic Data Length

We study with two settings of synthetic data length, i.e., from real input length 4K to target length 64K ($Ctx.=4K$) and from real input length 8K to target length 64K ($Ctx.=8K$).
Specifically, the chunk size $|\mathcal{C}_{i}|$ remains unchanged, while we set the number of chunks as 8 and 16 for the above two settings, respectively.
As shown in Fig. 6(a), short-context synthetic data length significantly diminishes the model’s performance on both the language modeling task and real-world tasks (data point $(M=2,\lambda=0.1,\mathit{Ctx.}=4K)$ versus data point $(M=2,\lambda=0.1,\mathit{Ctx.}=8K)$), but can still overcome the instruction tuning method (42.8 average score on LongBench) and effectively reduces the GPU memory requirement during training (Fig. 6(c)).
This is because when the original context length is relatively small (4K), it requires scaling up by a larger factor (16 times) to reach the desired context length (64K). During the positional indices synthesis process, some positional indices may miss or be infrequently activated, thereby impacting performance.

### 5.3 Comparison between SFT and LOGO

As shown in Fig. 7, we illustrate the impact of SFT (with two loss calculation strategies following (Xiong et al., )) and LOGO on the model’s generation and understanding performance throughout the training process.
We plot the trends of retrieval score (understanding ability) and recall score (generation ability) along the training progress.
We can observe that applying SFT loss to the entire sequence leads to a gradual decline in the LCM’s understanding ability, accompanied by performance fluctuations; while applying SFT loss solely to the prediction portion shows no significant improvement in model performance.
Nevertheless, applying LOGO can steer LCMs away from misaligned samples, thereby enhancing the recall score.
Simultaneously, it improves comprehension abilities, enabling the model to retrieve more key information within the context.

## 6 Conclusion

In this paper, we find that commonly used training approaches for LCMs may degrade the model’s generation capabilities, leading to misaligned outputs, such as hallucinations and instruction unfollowing.
To mitigate this issue, we introduce LOGO, a novel preference optimization training strategy for long-context alignment.
Specifically, LOGO has two key components: (1) a reference-free preference optimization objective that teaches the model to distinguish between the preference and the dis-preference predictions, and (2) a data construction pipeline tailored for the training objective, both of which are designed to ensure the training efficiency and effectiveness.
By performing LOGO training on a single 8$\times$A800 GPU machine within 16 hours, LCMs can achieve great improvements in long-context tasks while maintaining their inherent capabilities.
Besides, LOGO can also potentially scale the context length of short-context models and achieve better generation performance compared to other frequently used context scaling methods.

## Appendix A Limitation and Future Work

This paper presents an efficient preference optimization training strategy (LOGO) tailored for long-context alignment.
However, there are several limitations:

- •
Due to resource constraints within the academic community, the evaluation of real-world testing sets in LongBench may be affected by the varying prompts selected by different studies, which can lead to significant discrepancies in results. Consequently, we are unable to directly replicate the results from other works
- •
As mentioned in the main body (Sec. 3.2.2), there remains a lack of suitable evaluation models to assess whether the outputs of LCMs are accurate or contain hallucinations. The LOGO training objective proposed in this paper still has room for improvement.
- •
During the data construction phase, utilizing higher-quality datasets could yield better outcomes. However, as an academic paper, we believe we have demonstrated the generalizability of our method through the main experiments.

Moving forward, we plan to continue our research along the lines of efficient long-context alignment, particularly in algorithm development.
We aim to explore the integration of more effective evaluation strategies, such as RAG checkers (Ru et al., ), to assist in constructing preference and dis-preference instances. Additionally, we should investigate how to enhance the efficiency of our LOGO data construction pipeline across various tasks and domains.

In summary, this paper highlights the substantial potential of efficient training in long-context scenarios, and we hope our work will provide valuable insights for future research endeavors.

## Appendix B Details of Experiments in Introduction

In this section, we introduce the preliminary studies in the Introduction section, including the experimental settings, task definitions, and retrieval score calculation.

##### Experimental Settings

In Fig. 1(a) and Fig. 1(b), we evaluate the model performance on the subsets in LongBench (Bai et al., ), including Single Document QA, Multi-Document QA, Summarization, and Few-shot tasks.
For each long-context model, we utilize the same official instructions to guide the model prediction.

##### Multi-values Needle-in-a-Haystack

In Fig. 1(c), we calculate the retrieval score on the Multi-values Needle-in-a-Haystack dataset, which requires LCMs to recall multiple values within the context. We provide an example in Fig. 8:

Figure: Figure 8: Demonstration of Multi-values Needle-in-a-Haystack testing sample.

The formal definition of the task is as follows: Given $n$ questions $vq$ and its corresponding answers $K=\{vk_{j}\}_{j=1}^{n}$ (the needle), we insert $K$ in a synthetic context ${\bm{c}}$ (the haystack) at random position index ranges $P=\{vp_{i}\}_{i=1}^{n}$.
We then require the models to answer ${\bm{q}}$ based on the haystack with the inserted needle.
It is worth noting that ${\bm{q}}$ and $K$ are unique and irrelevant to the context,
ensuring that if an answer is correctly generated, it is indeed copied from the context, not from the model’s internal knowledge.

##### Calculation of Retrieval Score

Based on Wu et al. (), we define the retrieval score as the recall score of salient tokens located by retrieval heads.
To enhance comprehension, we manage to utilize familiar symbols and definitions that align closely with previous research.
Specifically, denote the current token being generated during the auto-regressive decoding process as $x$, and the attention score of a head as $\bm{a}\in\mathcal{R}^{|{\bm{c}}|}$.
In the task of Multi-values Needle-in-a-Haystack, an attention head $h$ is denoted as a retrieval head if it meets the following criteria:

- •
$x\in{\bm{k}}_{i}$, where ${\bm{k}}_{i}\in K$ and $x$ is a token within any one of the needle sentences in $K$.
- •
${\bm{c}}_{j}=x$, $j=\arg\max(\bm{a})$, $j\in{\bm{p}}_{i}$, ${\bm{p}}_{i}\in P$, i.e., the input token that receives the highest attention probability by this head is a token within any one of the needle in $K$ and is the same token as the currently generated token.

Let ${\bm{g}}_{h}$ be the set containing all copy tokens (also can be viewed as the located tokens) and pasted by a given head $h$, we define:

$$ $\text{Retrieval score for head}\;\;h=\frac{|{\bm{g}}_{h}\cap{\bm{k}}_{i}|}{|{ \bm{k}}_{i}|},$ (5) $$

It is worth noting that the retrieval score represents a token-level recall rate of the most attended tokens by an attention head.
After obtaining the retrieval score for each head, we start by filtering out the non-retrieval heads by setting the threshold at 0.1. This means that if a head performs copy-paste 10% of the time or more, it will be considered a retrieval head.
Then, we calculate the retrieval head score by averaging the scores of the top 10 attention heads from the remaining retrieval heads.

## Appendix C Design of LOGO Training Objective and Error Pattern Definition in LCMs

Misaligned predictions generated from LCMs can be specifically categorized into two types: failing to follow instructions and generating hallucinations.
In Fig. 9, we illustrate these two error patterns. Specifically, we define different error patterns by utilizing the degree of overlap between entities in the responses and the questions, along with specific templates:

- •
Instruction Unfollow: the entities in the model’s responses do not overlap with the entities in the question.
- •
Hallucination: there is a partial intersection of entities between the model’s responses and the question, and the entities in the response coincide with the main subject of the question.

It is worth mentioning that merely utilizing Named Entity Recognition (NER) models and rule-based methods proves inadequate for identifying these patterns.
Instead, a more robust evaluation involving strong LLMs such as GPT-4 or human assessment is required to accurately identify these patterns.
Consequently, in the design of the LOGO training objective, we do not confine to constructing cases with specific error patterns.
Therefore, instead of finding one dis-preference instance with a specific error pattern, we can expand the dis-preference space to push the model away from a range of possible dis-preference instances.

Figure: Figure 9: Demonstration and statistical analysis of different error patterns in long context tasks, where we have the following definitions of misalignment: (1) Instruction Unfollow: The entities in the model’s prediction are different from the entities in the question; (2) Hallucination: The entities in the prediction overlaps with the entities in the question, but the answer is incorrect.
Refer to caption: x8.png

## Appendix D Positional Indices Synthesis Details

We visualize the positional indices synthesis process in Fig. LABEL:fig:pos_synthesis.
Specifically, to ensure that the synthesized positional indices do not disrupt the original text’s semantic structure while maximizing the extended context size, we employ two different strategies for positional bias $\mathcal{B}$: Continuous Chunk Positional Indices Synthesis (Fig. LABEL:fig:pos_syn_a) and Sparse Chunk Positional Indices Synthesis (Fig. LABEL:fig:pos_syn_b).
For Continuous Chunk Positional Indices Synthesis, the positional bias within the same chunk is consistent. For instance, in the first chunk $C_{0}$, the positional bias $\{b_{0},b_{1},\cdots,b_{|C_{i}|}\}$ are the same value sampled from distribution $\mathcal{U}(1,(K-k)/N)$.
This ensures that the semantic structure within the chunk remains intact but can lead to sparse synthesized positional indices, as there will be significant gaps between the positional indices among different chunks.
Thereby, we propose Sparse Chunk Positional Indices Synthesis to fill these gaps, where each positional bias $b_{i}$ is sampled uniformly according to Equ. 4.
Considering that Sparse Chunk Positional Indices Synthesis might disrupt the semantic structure of the text, we set the ratio of data for Continuous Chunk Positional Indices Synthesis and Sparse Chunk Positional Indices Synthesis to 9:1 in actual deployment.

## Appendix E Case Study of LOGO Data

In this section, we provide the training samples built based on the LOGO training data construction pipeline as illustrated in Sec. 3.2.2.
We present the training samples in Fig. 11, Fig. 12, Fig. 13,
and Fig. 14, where the training data exhibits different error patterns (misalignments) in their dis-preference instances.

Figure: Figure 11: LOGO training samples: case1.

Figure: Figure 12: LOGO training samples: case2.

Figure: Figure 13: LOGO training samples: case3.

Figure: Figure 14: LOGO training samples: case4.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 6
presentation: 4
questions: What is the potential for scaling LOGO to models trained on diverse, multi-modal data? For example, long video VLM. I know that this might be hard to resolve in the rebuttal. This is just a discussion.
soundness: 4
strengths: 1. This work is the first to study long-context alignment. The topic and methods are both novel.

2. LOGO can extend the context window of short-context models, allowing for flexible adaptation across various LCM architectures.

3. Experiments on various benchmarks including needle in the hay-stack is promising.

4. The LOGO strategy effectively optimizes LCMs using limited data and resources, achieving comparable results with larger models.
summary: The paper introduces LOGO, a preference optimization training strategy to improve long-context alignment in language models. LOGO uses a reference-free preference objective and a position synthesis method to address memory constraints and efficiently train LCMs. With only 0.3B tokens on limited GPUs, LOGO achieves notable performance comparable to GPT-4 on real-world long-context tasks while preserving other model capabilities.
weaknesses: 1. If we use flash-attention (ring-attention) & deepspeed zero3 cpu offload, it is all right to train Llama-3-8B on 80k context (I already tested it). I think this should be a baseline to compare with the proposed Positional Indices Synthesis. The comparison should include both GPU memory, training hours and accuracy.

2. Would you please try longer context for evaluation? It seems that the longest context is commonly 80k in the paper, which might not be enough this year. For example, qwen2 models is commonly pre-trained as 128k context. It is able to train about 256k context with ring-attention (and the proposed Positional Indices Synthesis).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.How does LOGO fundamentally differ from DPO in handling long-context scenarios?

2.What theoretical guarantees can be provided for the position synthesis method?

3.How does the method scale with increasing context lengths beyond 32k tokens?

4.Can you provide detailed analysis of failure cases?
soundness: 3
strengths: -The paper proposes a new attention mechanism that combines the strengths of existing methods. This results in improved performance on long-range dependencies. It also enables efficient handling of long-context training with limited computational resources.

-The authors thoroughly evaluate their method on multiple benchmark datasets. They demonstrate its effectiveness in various settings and show a clear improvement over baseline methods.
summary: This paper presents a novel approach to long-context language modeling, leveraging a combination of attention mechanisms and position encoding to improve performance on long-range dependencies. The method shows promising results in improving long-context understanding while maintaining computational efficiency.
weaknesses: -The core idea of using preference optimization for long-context alignment seems like a straightforward extension of existing methods such as DPO and SLiC. The position synthesis method shows similarities to existing techniques like ALiBi and RoPE. The paper's main contribution appears incremental rather than transformative.

-The preference optimization objective (Equation 3) is similar to DPO without significant modification. The position synthesis method lacks theoretical justification for its effectiveness. The training procedure fails to address the fundamental challenges of long-context understanding.

-Experimental Limitations: While the authors compare their method to several existing approaches, the comparison is not exhaustive, and some relevant methods are not considered.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: How does LOGO compare with recent baselines such as [1], and methods included in your related work?
Please add comparsion with pipeline using long context and preference optimization, for example LongRoPE[2]&SimPO[3].
Since your contribution focus on long context alignment, please eval it on corresponding benchmark, LongAlign[3].
Could you provide more theoretical analysis such as error bounds for LOGO and analyze its convergence properties?

[1] Zhao, Hao, et al. "Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning." Forty-first International Conference on Machine Learning.
[2] Ding, Yiran, et al. "LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens." Forty-first International Conference on Machine Learning.
[3]Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." arXiv preprint arXiv:2405.14734 (2024).
[4] Bai, Yushi, et al. "Longalign: A recipe for long context alignment of large language models." arXiv preprint arXiv:2401.18058 (2024).
soundness: 3
strengths: Combining preference optimization with long-context alignment addresses a gap in current LCM training methods.
Develops a creative data construction pipeline that effectively creates preference/dis-preference pairs without requiring extensive human annotation
Clear experimental methodology with detailed ablation studies that validate design choices
Well-structured presentation with clear problem motivation and solution development
summary: This paper introduces LOGO, a novel training strategy that addresses the challenge of improving long context language models' (LCMs) generation capabilities while maintaining efficiency. While existing LCMs can effectively locate important information in long contexts, they often struggle with generating appropriate responses, leading to hallucinations and misaligned outputs. LOGO tackles this through a reference-free preference optimization approach that teaches models to distinguish between preferred and dis-preferred outputs, combined with an efficient data construction pipeline utilizing positional indices synthesis. The method's key advantage is its resource efficiency - requiring only 0.3B tokens of training data and 16 hours on a single 8×A800 GPU machine - while achieving comparable performance to GPT-4 on long-context tasks and maintaining performance on traditional benchmarks. The authors demonstrate LOGO's effectiveness across various tasks and its ability to extend context windows of existing models while enhancing their generation quality.
weaknesses: Lack of rigorous evaluation methods for detecting misaligned outputs and hallucinations, which affects the quality assessment of preference/dis-preference pairs
While the paper provides implementation details, the quality of training data could significantly impact results, and the paper uses relatively simple datasets
The theoretical justification for why preference optimization works better than traditional methods in long-context scenarios could be stronger

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	In the Preference and Dis-preference Data Synthesis section, you mentioned generating preferred data using πθ. Then, in the experimental section, you stated that you used long-llm-data as the training data. As far as I know, long-llm-data already includes standard answers. Did you generate additional answers using πθ beyond these standard answers? If so, what specific model was used as πθ—was it the policy model itself?

2.	You mentioned using long-llm-data as training corpus. To my understanding, this corpus, especially for the single-detail QA, multi-detail QA, and summarization datasets, was already instruction-tuning dataset. So, why do you mention at the end of the Evaluation Settings part that 12,000 data samples from LongAlpaca were used as instruction training data?

3.	Compared to using standard instruction tuning on long-llm-data, how much additional performance improvement does the SimPO loss provide? As far as I know, simple instruction tuning on long-llm-data already yields strong performance on LongBench.
soundness: 2
strengths: 1. Innovative Training Strategy The introduction of LOGO, a long-context alignment method combined with preference optimization, improves the generation capabilities of LCMs.

2. Efficient Training LOGO adopts a position index synthesis method, allowing training to be completed with limited data and resources (8×A800 GPUs on a single machine in 16 hours), significantly improving training efficiency.

3. Significant Performance Improvement In real-world tasks, the Llama-3-8B-LOGO model significantly outperforms GPT-3.5-Turbo and approaches the performance of some top-tier closed-source models like GPT-4, while maintaining strong performance in short-context tasks as well.
summary: This paper presents a novel preference alignment method for long-text models, combining position encoding expansion with human preference alignment techniques. For position encoding expansion, the authors propose splitting ultra-long contexts into multiple chunks, applying continuous position encoding within each chunk, and using a jump-based position encoding between chunks to achieve extended position encoding. In terms of preference alignment, the authors generate responses of varying quality by providing different qualities of context, treating higher-quality responses as preferred and lower-quality ones as non-preferred. These are then fed into SimPO for preference learning. Beyond SimPO loss, the authors also incorporate a weighted language modeling loss into the total loss.

Thanks to this unique position encoding expansion approach, the language modeling loss corresponding to strongly relevant contexts is not overly smoothed, thus improving optimization efficiency while reducing issues such as hallucination. On the other hand, the introduction of the powerful SimPO further strengthens the model’s instruction-following ability.
weaknesses: More controlled experiments should be conducted, comparing the performance of models under the same experimental conditions: (1) using only instruction tuning, (2) using instruction tuning + SimPO (with SimPO’s positive and negative samples that already exist in the training corpus, rather than those generated by policy models or other LLMs), and (3) using the full LOGO method. These comparisons would clarify that the effectiveness of LOGO is not solely attributable to either instruction tuning alone or to the straightforward combination of instruction tuning and SimPO.

Meta-review: (a) The paper introduces LOGO, an efficient preference optimization method for long-context alignment using a reference-free approach and positional index synthesis. The model achieves competitive results with GPT-4 while requiring limited resources.

(b) Strengths include computational efficiency, novel preference optimization, and strong experimental results on long-context tasks.

(c) Weaknesses include incremental contributions, insufficient theoretical justification, and limited evaluation for other baselines or tasks. Some reviewers flagged the need for a stronger analysis of misalignment and scalability.

(d) While promising, the incremental nature and unresolved concerns lead to rejection. The paper lacks rigorous baselines and theoretical depth.

Decision: Reject