Selecting Influential Samples for Long Context Alignment via Homologous
  Models' Guidance and Contextual Awareness Measurement
Authors: name: Shuzheng Si; name: Haozhe Zhao; name: Gang Chen; name: Yunshui Li; name: Kangyang Luo; name: Chuancheng Lv; name: Kaikai An; name: Fanchao Qi; name: Baobao Chang; name: Maosong Sun
Published: 2024-10-21 04:30:53

Abstract: The expansion of large language models to effectively handle instructions
with extremely long contexts has yet to be fully investigated. The primary
obstacle lies in constructing a high-quality long instruction-following dataset
devised for long context alignment. Existing studies have attempted to scale up
the available data volume by synthesizing long instruction-following samples.
However, indiscriminately increasing the quantity of data without a
well-defined strategy for ensuring data quality may introduce low-quality
samples and restrict the final performance. To bridge this gap, we aim to
address the unique challenge of long-context alignment, i.e., modeling the
long-range dependencies for handling instructions and lengthy input contexts.
We propose GATEAU, a novel framework designed to identify the influential and
high-quality samples enriched with long-range dependency relations by utilizing
crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
(CAM). Specifically, HMG attempts to measure the difficulty of generating
corresponding responses due to the long-range dependencies, using the
perplexity scores of the response from two homologous models with different
context windows. Also, the role of CAM is to measure the difficulty of
understanding the long input contexts due to long-range dependencies by
evaluating whether the model's attention is focused on important segments.
Built upon both proposed methods, we select the most challenging samples as the
influential data to effectively frame the long-range dependencies, thereby
achieving better performance of LLMs. Comprehensive experiments indicate that
GATEAU effectively identifies samples enriched with long-range dependency
relations and the model trained on these selected samples exhibits better
instruction-following and long-context understanding capabilities.