Selecting Influential Samples for Long Context Alignment via Homologous Models’ Guidance and Contextual Awareness Measurement

Authors: Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, Maosong Sun
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Long context alignment, Large language models, Data selection, Efficient instruction tuning

Abstract:
The expansion of large language models to effectively handle instructions with extremely long contexts has yet to be fully investigated. The primary obstacle lies in constructing a high-quality long instruction-following dataset devised for long context alignment. Existing studies have attempted to scale up the available data volume by synthesizing long instruction-following samples. However, indiscriminately increasing the quantity of data without a well-defined strategy for ensuring data quality may introduce low-quality samples and restrict the final performance. To bridge this gap, we aim to address the unique challenge of long-context alignment, i.e., modeling the long-range dependencies for handling instructions and lengthy input contexts. We propose GATEAU, a novel framework designed to identify the influential and high-quality samples enriched with long-range dependency relations by utilizing crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement (CAM). Specifically, HMG attempts to measure the difficulty of generating corresponding responses due to the long-range dependencies, using the perplexity scores of the response from two homologous models with different context windows. Also, the role of CAM is to measure the difficulty of understanding the long input contexts due to long-range dependencies by evaluating whether the model’s attention is focused on important segments. Built upon both proposed methods, we select the most challenging samples as the influential data to effectively frame the long-range dependencies, thereby achieving better performance of LLMs. Comprehensive experiments indicate that GATEAU effectively identifies samples enriched with long-range dependency relations and the model trained on these selected samples exhibits better instruction-following and long-context understanding capabilities.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730521442075

thread_type:
official_review

<thread_object>
creator: reviewer_93ru

<content>
summary:
This paper introduces GATEAU, a data selection framework designed to identify critical instruction samples for long context alignment of LLMs. Specifically, GATEAU consists of two core modules: Homologous Models' Guidance (HMG) and Contextual Awareness Measurement (CAM). HMG assesses the difficulty of generating responses due to long-range dependencies by comparing the perplexity scores of homologous models with varying context windows. CAM measures how effectively a model focuses on relevant segments of extended input. The outputs of HMG and CAM are combined to produce a final ranking criterion for data selection. Extensive experiments indicate that the proposed method significantly enhances the long-context instruction-following capabilities of LLMs.

strengths:
1. This paper is well motivated. Identifying high-quality long instruction-following samples is essential for improving long context alignment. 
2. The proposed GATEAU aligns naturally with the challenges of long-range dependency modeling, making the methodology intuitive and easy to understand.
3. The authors demonstrate the effectiveness of GATEAU through extensive experiments across multiple benchmarks. The results consistently show that the selected samples significantly improve model performance on both long and short instruction-following tasks.

weaknesses:
1. The proposed method may assign similar high scores to duplicate or highly similar samples, assuming they contribute independently to model improvement. However, repeated exposure to similar samples may not add incremental value and could even undermine the alignment process of LLMs.
2. The paper does not discuss the computational resources required to implement the GATEAU framework. A selection method that is slow or computationally prohibitive is less feasible in practical scenarios, especially for extremely long instruction-following examples.
3. Although the paper includes ablation studies, it lacks an experiment testing the effectiveness of using a single model's perplexity (e.g., LLaMA-2-7B-base-64k) as the sole criterion for data selection. Adding this experiment would provide valuable insight into the authors' claim on page 4, line 174, that high perplexity alone does not adequately reflect response difficulty in long-context scenarios.

questions:
1. How does the proposed framework handle potentially redundant or highly similar samples? Could repeated high-scoring samples lead to an overrepresentation of certain types of data, thus limiting the diversity and richness of long-context dependencies in the selected dataset?
2. Would this approach remain feasible for datasets of larger scale, and what are the expected costs in terms of time and resources?
3. Is there some display problem in Figure 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582268516

thread_type:
official_review

<thread_object>
creator: reviewer_kged

<content>
summary:
The paper introduces GATEAU, a framework aimed at improving the alignment of large language models with long-context data by identifying influential samples. GATEAU leverages two main components: Homologous Models’ Guidance, which assesses the difficulty of generating responses based on perplexity differences between models with varying context windows, and Contextual Awareness Measurement, which evaluates whether models focus on crucial segments in long input contexts. Through these mechanisms, GATEAU selectively curates samples to enhance LLMs' ability to follow instructions with long-range dependencies.

strengths:
- The paper addresses an important challenge of optimizing data selection for long-context alignment, enhancing LLMs' performance in real-world applications that require handling long, complex contexts.
- The paper provides comprehensive evaluations across multiple tasks and varied compression ratios, which helps illustrate the model’s versatility and effectiveness.
- The methodology for Homologous Models’ Guidance is well-motivated. Leveraging LLM collaboration for data selection is interesting.
- Ablation study and human evaluation are conducted to verify the effectiveness of the method.

weaknesses:
- The study only tests on the LLaMA2 family, which may restrict the generalizability of findings. Additionally, HMG relies on perplexity differences between homologous models with different context windows, leaving unclear guidance on applying the technique to other models lacking such variants.
- There is no analysis of the time efficiency of the data selection method. Since CAM requires estimating importance scores for each segment, which may be computationally expensive, time efficiency is a critical factor.
- The CAM module’s process for calculating attention weights is vague, particularly in obtaining the attention weight across tokens in response $y$ to token $t_j$. Including pseudo-code or further clarification would make the implementation more accessible.
- The method of dividing long inputs into equal segments may overlook the fact that important information could span multiple segments. Additionally, segment importance can depend conditionally on other segments, an aspect not accounted for in the current approach.
- The method averages attention across heads and layers without distinguishing them, potentially introducing noise, as different heads or layers may capture varied aspects of the input [1]. Calibrating the attention aggregation could improve focus on relevant input sections.
- Key hyperparameters, such as the number of segments, remain unexplored, missing potential for optimization in specific contexts.

[1] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. ICLR 2024

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730984533875

thread_type:
official_review

<thread_object>
creator: reviewer_pvw2

<content>
summary:
This paper proposes a new data selection algorithm, GATEAU, for training long-context instruction following models. Gven a long-context instruction following dataset, the algorithm calculates two metrics: Homologous Models' Guidance (HMG) and Contextual Awareness Measurement (CAM). In particular, HMG measures the difficulty of generating responses due to long-range dependencies by by comparing perplexity scores of responses between two homologous models with different lengths of context windows. CAM measures the difficulty of  utilizing important parts of the long-input by evaluating whether the impoartant segments are being utilized using the attention scores. The author measures the effectiveness of the proposed method on various benchmarks (e.g., LongBench, LongBench-Chat and MT-Bench).

strengths:
1. The writing is clear (except some questions I have in the Questions section below) and the author explains the proposed method well with clear formulations.
2. The author uses different benchmarks (LongBench, LongBench-Chat, MT-bench, Needle-in-a-haystack) to show the effectiveness of the proposed method on (long/short)-context instruction following tasks.

weaknesses:
1. 
- The method proposed heavily depends on the perplexity that measures the similarity between the base model's answer and the desired answer. The model $\theta_A$ (e.g., LLaMA-2-7B-base-64k as mentioned in the paper) in the method that calculates the perplexity is a base model (model that only does completion). It does not make sense to measure the perplexity of a base model on an instruction-following dataset because the base model wasn't trained to follow instruction. In this case, the perplexity can be high and it does not accurately measures the difficulty of a document. The author also acknowledges this and in the first metric (HMG) uses a homologous short context model $\theta_B$ as a reference model and calculates the difference between PPL of $\theta_A$ and $\theta_B$. The author claims that this "mitigate the influence brought by lacking other capabilities" (e.g., instruction-following capability, long-context capability). However, there is no guarantee that lacking other capabilities contribute equally in increasing PPL of two models. Essentially the data where the models measure the PPL on is out-of-distribution data (both models were not trained on the instruction-following data and $\theta_B$, which is extended to a longer-context window using zero-shot method, was not trained on long-context data) and model's behavior is unpredictable. Plus, $\theta_A$ was obtained by continual pretraining on long-context data. While HMG assumes $\theta_A$ and $\theta_B$ are similar models, it also depend on what the continual pretraining dataset is (here the method implicitly assumes that the additional dataset is small). If the additional dataset is very large, $\theta_A$ and $\theta_B$ can perform very differently.
- Moreover, for second metric (CAM), the LLM that measures the PPL was not trained on instruction-following dataset (and therefore the behavior is unpredictable) and there is no such reference model.

2. LESS [1] is an optimizer-based method that select a subset of instruction-following dataset by estimating data influence (selecting data points that minimizes the validation loss) and this can be as one of the baselines.

Minor issues:
1. For Table 5, current bold numbers are a bit misleading. Usually bold numbers indicate the highest numbers across some category but here it seems the proposed method is bold. Also, for 13B models, it seem the w/o HMG and w/o CAM settings are not reported. Is there a particular reason of not doing so (e.g., computational constraint)? The ablation study on 7B model does show the effectiveness.

[1] Xia, M., Malladi, S., Gururangan, S., Arora, S., & Chen, D. (2024). Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333.

questions:
1. For perplexity guidance (line 377), which $\theta$ do you use ($\theta_A$ or $\theta_B$ -- I am assuming $\theta_B$ here)?
2. What is the context length of MT-Bench? The paper mentions that MT-Bench is for short-context instruction following. Since the proposed method, GATEAU, is designed for long-context, do you have any hypothesis on why it also improves on the short-context instruction following tasks (Table 4)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731287882406

thread_type:
official_review

<thread_object>
creator: reviewer_rtww

<content>
summary:
To effectively handle instructions with extremely long contexts in expansion of large language models (LLMs), this paper proposes GATEAU, a novel framework designed to identify the influential samples enriched with long-range dependency relations by utilizing crafted Homologous Models’ Guidance (HMG) and Contextual Awareness Measurement (CAM). Specifically, HMG uses the perplexity scores of the response from two homologous models to measure the ability in modeling long-range dependencies. CAM is used to measure difficulty of understanding the long input contexts by evaluating whether the model’s attention is focused on important segments. Extensive experiments on several LLM benchmarks demonstrate the effectiveness of the proposed algorithms.

strengths:
(1)	This paper proposes an efficient and practical influential-sample-selecting algorithm for long context alignment of large language model. The motivation of the HMG and CAM component is clearly explained in Section 3.

(2)	The organization of this paper is clear and easy to follow. The notations used in Section 3 are all well clarified.

(3)	The experiments are truly extensive, validating the effectiveness of the proposed methods. In particular, the SOTA result in Tables 1-4 is a surprise for me that using part (e.g. 10\%) of the whole dataset can achieve better performance than using the whole dataset. Ablation study in Section 4.3 also verifies the effectiveness of the components (i.e. HMG and CAM).

(4)	Overall, the effort in selecting influential samples for long context alignment in LLM truly makes sense, and the proposed algorithm is simple, motivated, and effective.

weaknesses:
(1)	In Eq.(2), the explanations for the new notation $\theta _{A}$ and $\theta _{B$ should follow immediately after their appearance in Eq.(2).

(2)	In Tables 1-4, the captions should be put at the top (instead of the bottom) of the table.

(3)	In section 4.2 Impact of GATEAU, the paper first analyzes the experimental results in Tables 2 and 4 (line 413), and then analyzed the results in Tables 1 and 3 (line 429). It will be better to analyze the experimental results according to the order of Tables.

(4)	The memory storage and running time of the proposed algorithm is missing, which in my opinion can help readers understand the proposed algorithms more comprehensively. For example, in Eq.(2), HMP model uses short context model $\theta _{A}$ and long context model $\theta _{B}$ to compute the perplexity distance, so does the proposed algorithm have more model parameters (e.g. of  $\theta _{A}$ and $\theta _{B}$ ) than the existing algorithm (e.g. only has $\theta _{B}$)? Therefore, I would suggest to add a table to compare the memory burden as well as the execution time with the existing methods on long context alignment.

(5)	It is a surprise for me that in Table 1 and 3, the proposed algorithm with less dataset can achieve SOTA results in all settings. So, can you give some example (whether simulation or real-world dataset, or theoretical analysis) to explain in what kind of situation the proposed algorithm may fail to achieve SOTA results?

(6)	If I understand correctly, in the proposed algorithm, we need to choose some part of the whole dataset (e.g. 10\% or 30\%) to train the model. How does this paper choose this percentage? And in real-life applications, how do we choose the part (whether 10\% or 30\%) from the whole dataset to align LLM?

questions:
(1)	Could you give some comparisons of memory burden between the proposed algorithms and other exiting methods?

(2)	Could you give some examples to show the failure of the proposed algorithm? It is a little unbelievable that one algorithm can achieve the best performance on all datasets, especially with less training data than existing ones.

(3)	Could you give some insights into the setting of the utilization percentage of the whole dataset of the proposed algorithm?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732344067994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
Dear Reviewers,

We thank all reviewers for their insightful comments and acknowledgment of our contributions.

**We greatly appreciate your recognition of the strengths of our work as follows:**

- **Introduction of GATEAU**
    - We present the GATEAU, **as a novel framework(`rtwW`)**, recognized by **all reviewers as well-motivated.**
    - Our framework offers an **efficient and practical** algorithm (**`rtwW`**) to address an **important and essential** challenge(**`KGEd`**,**`93rU`**) for long-context alignment.
- **Methodological Effectiveness**
    - Our method has been acknowledged by **all reviewers** for its **effectiveness, with a clear motivation(`rtwW`,`KGEd`),** which naturally aligns with the challenges of modeling the long-range dependency(**`93rU`**).
- **Comprehensive Experiments**
    - Our Experiments are acknowledged for their **comprehensiveness** by **all reviewers**, validating the **versatility** and **effectiveness** of the proposed methods(**`KGEd`**, **`93rU`**, **`rtwW`**), and showcasing a **well-motivated and interesting**(**`KGEd`**) method for this field.

**We've revised our manuscript per the reviewers' suggestions** (**highlighted in red** in the uploaded revision pdf). Detailed responses to each reviewer's concerns are carefully addressed **point-by-point**.

Below summarize the **major updates** we've made:

- **Presentation:**  We fix the typos in the uploaded revision pdf to make the organization of this paper clearer according to the suggestions from reviewers **`rtwW`** and **`pvW2`**.
- **Experiment:** We further conduct the following experiments to make our paper more sound and try to address reviewers' concerns.
    - **Additional ablation study on 13B models**: (**`pvW2`**). We further conducted the **additional ablation study for 13B models** in our revised paper. This indicates the effectiveness of our proposed GATEAU.
    - **Comparison of execution time**: (**`rtwW`,** **`pvW2`**, and **`93rU`**). We further **compare** the **execution time** of our method with other baselines. The experiment shows that our method introduces **acceptable offline time** overhead compared to the **other baselines** and the SFT stage. Meanwhile, the performance results of our proposed GATEAU (consists of HMG and CAM) demonstrate that the additional execution time is worthwhile.
    - **Exploration of the diversity of selected samples**: (**`93rU`**). We explore the **diversity** of the samples selected by GATEAU. Results show that our proposed GATEAU has partially **addressed sample redundancy** and implicitly **ensured the diversity** of selected long SFT data.
    - **Further exploration of HMG**:(**`pvW2`**) We further explore the designed HMG. As shown in experiments in our response to the **reviewer `pvW2`**, we find that using only base models in the HMG phase can achieve good results. It shows that **HMG can effectively minimize the influence of other factors** (e.g., the limited instruction-following ability of a base model) and model the difficulty in modeling the long-range dependencies.
    - **Further exploration of CAM**: We explore the effect of the number of segments in CAM based on the review from **`KGEd`**. As shown in our experiment **in our response to the reviewer `KGEd`**, different segment lengths affect the model's performance; however, as long as a reasonable length value is chosen, the fluctuations in model performance are not significant. Meanwhile, the performance will always be improved over using the whole long SFT dataset and only using the HMG method, showing the effectiveness of our proposed CAM.
- **Explanation:** We attempt to provide the following explanation to address the misunderstandings in our paper.
    - **GPU memory burden:** We discuss the process of our proposed GATEAU in detail, and explain our proposed GATEAU does **not add an extra GPU memory** burden.
    - **Need for using a single model's perplexity as the sole criterion for data selection:** For the concern from reviewer **`93rU`**, we have already compared this important baseline in the submission version of our paper, namely **Perplexity Guidance**, in our comprehensive experiments (e.g., **Table 1, Table 2, Table 3, Table 4, and Table 8 in the Appendix**) to show our valuable insight, i.e., the high perplexity alone does not adequately reflect response difficulty in long-context scenarios.

We believe our work could make a novel contribution to the community and offer a novel perspective on addressing the challenges of long context alignment of LLM.

We would like to be involved in further discussions if any question is raised.

Best,

Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549993415

thread_type:
official_comment

<thread_object>
creator: area_chair_ruae

<content>
title:
Please participate in the discussion with the authors

comment:
Dear Reviewers,

Thank you for your efforts and contribution to ICLR! The authors have posted their responses to your original comments. Only less than two days are left for the reviewer-author discussion. Given the current borderline ratings, your help and prompt responses are important. Please actively check the authors' responses and participate in the discussion. 

Thanks!

Best regards,
Your AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310023398

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Reviewers to Participate in the Discussion

comment:
Dear Reviewers,

As today is the last day of the discussion, we would like you to take a look at our responses and kindly reconsider the ratings. Thanks.


Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview: