Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models (2024)

Abstract: Long-context modeling capabilities are important for large language models (LLMs) in various applications. However, directly training LLMs with long context windows is insufficient to enhance this capability since some training samples do not exhibit strong semantic dependencies across long contexts. In this study, we propose a data mining framework ProLong1 that can assign each training sample with a long dependency score, which can be used to rank and filter samples that are more advantageous for enhancing long-context modeling abilities in LLM training. Specifically, we first use delta perplexity scores to measure the Dependency Strength between text segments in a given document. Then, we refine this metric based on the Dependency Distance of these segments to incorporate spatial relationships across long contexts. Final results are calibrated with a Dependency Specificity metric to prevent trivial dependencies introduced by repetitive patterns. Moreover, a random sampling approach is proposed to optimize the computational efficiency of ProLong. Comprehensive experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies, and LLMs trained on these documents exhibit significantly enhanced long-context modeling capabilities.

heading: 1 Introduction
text: Large language models (LLMs) are widely used in many natural language processing (NLP) tasks (Brown et al., 2020). These tasks often require dealing with long text inputs (Bai et al., 2023; Zhang et al., 2023), such as lengthy documents (Zhou et al., 2022), long conversation histories in chatbots (Zhong et al., 2023) or large codebases (Guo et al., 2023). Therefore, enhancing
*Equal contribution. †Corresponding author. 1 https://github.com/October2001/ProLong
LLMs to model long-context inputs is a prominent desiderata.
There are primarily two categories of approaches to expand the context window of an LLM. The first category fine-tunes LLMs with longer context windows (Chen et al., 2023a), while the second category adjusts the LLM’s positional encoding or attention mechanism to accommodate larger position indices without further re-training (Press et al., 2021; bloc97, 2023). However, non-training methods often produce results inferior to those of finetuned LLMs (Xiong et al., 2023), which model long-context inputs more effectively and generally achieve lower perplexity scores.
Although reported to be feasible, simply finetuning LLMs with naively sampled long corpora does not ensure improved long context modeling capabilities (de Vries, 2023). Some of these finetuned LLMs may still struggle to effectively process and utilize information from long input contexts even if they obtain a decently low perplexity score (Sun et al., 2021; Pal et al., 2023). This can lead to low performance in various downstream applications, even in some basic synthetic retrieval tasks (Liu et al., 2023). Nevertheless, few approaches try to tackle this long-context modeling
8222
issue from a data-centric perspective. As revealed by Fu et al. (2023), the quality of fine-tuning data plays a critical role in enhancing the long-context modeling capabilities of LLMs (Li et al., 2023b). Besides, de Vries (2023) also report that high-quality corpora significantly outperform other factors in boosting long-context performance. Upon further exploration, we recognize that high-quality long-text data is characterized by the presence of long-range dependencies. The importance of encapsulating long-range dependencies in LLMs is also underscored by Borgeaud et al. (2022), which elucidates the benefits of integrating global dependencies into retrieval-augmented language models.
However, such strong semantic dependencies are rare in typical training samples (de Vries, 2023) and diminish as the distance between segments increases (Staniszewski et al., 2023). Even with identical sequence lengths, different samples may exhibit varying dependency density. Specifically, certain long training samples often comprise concatenated short documents that are randomly selected and do not have any semantic dependencies. Moreover, even for inherently long documents, like novels, most tokens depend only to a brief span of preceding context. This phenomenon can be simply concluded as “long context is not long at all”, leading to challenges in model learning (Figure 1). Therefore, we argue that explicitly incorporating long-dependency data into the fine-tuning process can facilitate long-context modeling.
In this paper, we propose a novel framework (called ProLong) to mine long-dependency data. ProLong assigns a long-dependency score to each document, which serves as an indicator of the dependency density across long contexts. Documents with higher scores are deemed more advantageous for boosting long context modeling, therefore we can use these scores to rank and filter high quality corpus for LLM fine-tuning. Concretely, ProLong first partitions each document into fixed-length segments and evaluates dependency relationships between each segment pair from three perspectives: (i) dependency strength quantifies the difference in perplexities of a given segment when conditioned with or without its preceding segments. This metric measures the contribution of the preceding segment to the current one; (ii) dependency distance measures the positional gap and spatial relationship between two text segments; and (iii) dependency specificity employs entropy to ensure a
non-uniform distribution of dependency strengths across all preceding segments, mitigating trivial dependencies introduced by repetitive patterns. A dependency score is assigned to each segment pair by combining the above three perspectives, and a final long-dependency score for the entire document is computed by accumulating dependency scores of all segment pairs.
Further, we adopt various strategies to optimize the computational efficiency of ProLong, including sampling among segments, evaluating perplexity scores with small models and curating test sets for rapid validation. Experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies and LLMs trained on these documents exhibit significantly enhanced long-context modeling capabilities. Our contributions are summarized as follows:
1. To the best of our knowledge, this is the first study to explore the relationship between dependency density and the quality of long-text data.
2. We propose ProLong, a data mining framework for identifying long-dependency data. With ProLong, significant performance boosts are observed using only 50% of fine-tuning data.
3. We provide an in-depth analysis of ProLong’s components, optimizing computational efficiency and making it practical for large-scale corpora.
4. We develop two models, ProLong-7b/13b, using training samples derived from the ProLong framework. Experiments show that our models outperform equal-sized competitors on both language modeling and real long-context tasks.

heading: 2 Related Work
text: Long-context LLMs. Large Language Models (LLMs), such as Llama (Touvron et al., 2023a) with a context size of 2048 and Llama2 (Touvron et al., 2023b) with 4096, are typically pre-trained using a pre-determined context size. However, training LLMs from scratch with extended contexts is extremely time-consuming and labor-intensive. Therefore, recent research endeavors to extend the limited context length of these models via finetuning. Fine-tuning methods such as Positional Interpolation (Chen et al., 2023a), "NTK-aware" interpolation (bloc97, 2023), Giraffe (Pal et al., 2023) and YaRN (Peng et al., 2023) modify rotary position encoding (RoPE) (Su et al., 2024) and then fine-tune the model with a small number of training steps to expand the context window. De-
spite this, full fine-tuning is still computationally expensive. Consequently, researchers have pivoted towards exploring methods to reduce training costs. For instance, LongLora (Chen et al., 2023b) proposes S2-Attn and utilizes LoRA (Hu et al., 2021) for low-cost and efficient training; Soaring (Zhang et al., 2024) introduces Activation Beacon, achieving a dramatic extension of LLMs context at low training costs. Nonetheless, these methods have not taken into account the quality of the long-context data used during fine-tuning stage. In contrast, our approach emphasizes the high-quality long-context training data, facilitating the efficient extension of LLMs’ context with a limited quantity of data.
Constructing Long-context Training Data. Long-context data is not simply the arbitrary concatenation of unrelated short texts. The construction of high-quality or specialized forms of longcontext data also holds significant research value. de Vries (2023) argues against squandering attention on randomly connected long texts and instead proposes considering strategies for obtaining meaningful long-context data. SPLICE (Staniszewski et al., 2023) introduces structured packing for longcontext, constructing training examples by assembling multiple similar documents selected through a retrieval method. Ziya-Reader (Junqing et al., 2023) constructs a tailored Multi-doc QA task that requires concentration on different positions in contexts to address the "lost in the middle" (Liu et al., 2023) problem. Yu (2023) delves into the required data format for training long-context models, claiming that the use of "W-shaped data" is essential to tackle the "lost in the middle" problem. Upon further exploration, we identify that high-quality long-text data is characterized by the presence of long-range dependencies. Our method focuses on selecting these long-dependency data from a substantial corpus of long-context data.

heading: 3 ProLong Framework
text: Figure 2 presents the overall framework of ProLong, which assigns a Long Dependency Score (LDS) for a given data sample S (S could be an intact document or a concatenated training sample). Specifically, we first divide S into N segments of equal length c1, ..., cN . Then for each segment pair (cj , ci), j < i, three scores are calculated: 1. The Dependency Strength (DST) score evaluates how much semantic dependencies are there between ci and cj ; 2. The Dependency Distance (DDI) score
demonstrates the distance between ci and cj ; 3. The Dependency Specificity (DSP) score penalizes the erroneous gains introduced by similar patterns in each segments. The final LDS for S is obtained by merging the above three sets of scores.

heading: 3.1 Dependency Strength
text: For a given segment pair (cj , ci), j < i, the dependency between ci and cj can be evaluated by the delta perplexity score of ci when cj is given or not in the context. Specifically, if there are strong semantic dependencies between ci and cj , then it is expected that the perplexity score of ci will be dramatically reduced if cj is presented in the context. Therefore, we can select a language model and use the difference of perplexity scores to quantify the dependency strength DSTi,j between ci and cj :
DSTi,j = PPL (ci)− PPL (ci | cj)
PPL (ci) (1)
where PPL (ci) is the perplexity score of ci without any context, and PPL (ci | cj) is the conditional perplexity score of ci when cj is provided in the input context. In this study, we use a small-sized and fixed model to calculate perplexity scores.

heading: 3.2 Dependency Distance
text: Another factor to consider in long dependency relationships is the distance between ci and cj . Specifically, distant segments are more important for learning long-range dependencies. Therefore for each pair (cj , ci), j < i, we consider a dependency distance score DDIi,j to measure the positional gap:
DDIi,j = i− j N − 1 (2)
where N is the total segment count.

heading: 3.3 Dependency Specificity
text: Our early experiments observe that some documents in the training corpus contain repeated text spans. Specifically, in the extreme case, the entire document contains only one unique token. For a given segment pair (cj , ci), j < i from these documents, ci and cj are nearly the same and ci will receive nearly perfect conditional perplexity score if cj is presented in the context. In this case, we will achieve an extremely high dependency strength score DSTi,j between ci and cj .
However, such trivial dependencies brought by repeated text spans are usually harmful to the training of LLMs (Lee et al., 2021). Therefore, we introduce the concept of dependency specificity (DSP)
to mitigate this issue. Specifically, for a given segment ci, we can calculate the reduction of perplexity scores ∆PPLi,j = PPL(ci)− PPL(ci|cj) when conditioning ci on each of its preceding segments cj , (j = 1, ..., i − 1). A probability distribution P (ci) = (p1, ..., pi−1) can then be obtained by applying Softmax on ∆PPLi,j :
pj = exp (∆PPLi,j)∑i−1 k=1 exp (∆PPLi,k)
(3)
P (ci) denotes the dependency distribution of ci on all preceding segments. A uniform P (ci) indicates that ci relies almost equally on each preceding segment cj (j = 1, ..., i − 1), suggesting that cj are nearly identical. Consequently, the entropy of P (ci) can serve to adjust the DST score derived from Eq.1, addressing the issue of repeated spans.
In this study, we define the dependency specificity DSPi score for a given segment ci as
DSPi = Emax − E
Emax (4)
where Emax = log (i− 1) is the entropy of a i− 1 dimensional uniform distribution. And E is the entropy of P (ci), i.e., E = − ∑i−1 j=1 pj log (pj). A higher DSPi value suggests that the segment ci does not depend equally on all its preceding segments, indicating a lower likelihood of these segments being repeated spans.

heading: 3.4 Long Dependency Scores
text: We define a long dependency score between each segment pair (cj , ci), j < i as:
LDSi,j = (αDSTi,j + βDDIi,j) ·DSPi (5)
where α, β are hyper-parameters that control the impact of DSTi,j and DDIi,j , respectively. Note that the formulation of Eq.5 is essentially an adjustment of the DSTi,j score. Concretely, if ci and cj exhibits strong semantic dependency over extended contexts, it is expected that ci and cj are spatially far (i.e., with large DDIi,j) and achieve a high DSTi,j , and DSTi,j attributed to genuine semantic connections rather than repeated spans (as indicated by high DSPi).
Totally, we define the long-dependency score for the entire sample S by accumulating LDSi,j :
LDS = N∑
i=1
i−1∑
j=1
LDSi,j · Ii,j (6)
Ii,j = { 1, if DSTi,j > τ 0, otherwise
(7)
where τ is a threshold for DSTi,j to filter out noises in the perplexity calculation.

heading: 3.5 Enhancing Computational Efficiency
text: The time complexity for calculating LDS in Eq.6 is O(N2), which is impractical when handling massive candidate samples. To optimize the computational efficiency of LDS, we employ a method of random sampling. First, we randomly sample T index pairs: D = {(x1, y1), ..., (xT , yT )}, where xt ∈ [1, N ], yt ∈ [1, N ] and xi < yi for t = 1, ..., T . Each element (xt, yt) ∈ D corresponds to a segment pair (cxt , cyt). Then, a sampled LDS is calculated as:
LDSsp = ∑
(xt,yt)∈D LDSxt,yt · Ixt,yt (8)
The computational of LDSsp is substantially decreased, lowering the time complexity to O(T ).

heading: 4 Experimental Setup

heading: 4.1 Training Datasets
text: Our training datasets consist of three parts: pretrain dataset, English book dataset, and code dataset. The pre-training dataset is sampled from RedPajama (Computer, 2023) to mitigate catastrophic forgetting induced by extended training. The pre-training dataset part remains the same across all experiments. We choose English books and code data primarily for extending training because they naturally possess sufficiently long lengths. Note that the length of each document in all datasets exceeds 32k. Detailed training data information is provided in Appendix A.1.

heading: 4.2 Task Formulation
text: Assume that there are multiple models based on different training datasets and methods. We comprehensively evaluate the long context capability of the models, including three tasks as follows:
Language Modeling Tasks. Language modeling task is the most fundamental requirement for LLMs, typically measured by perplexity (PPL) on the test text data. We follow Chen et al. (2023b) to select 128 documents that are randomly sampled from the total proof-pile (Azerbayev et al., 2022) test split. For each document, it has at least 32768 tokens. All PPL results are calculated using the sliding window method (Press et al., 2021) with stride S = 1024.
Synthetic Long Context Tasks. The synthetic long context tasks are divided into two sub-tasks: multi-document question answering (MQA) and key-value retrieval. Following the methodology of Liu et al. (2023), we make controlled changes to the input context size and the position of the relevant information within this context in both sub-tasks. In the MQA task, the total number of documents is either 20 or 30, corresponding to text lengths of 4k or 6k tokens, respectively. For the key-value retrieval task, we use a total of 140 or 300 key-value pairs, corresponding to total token counts of 8k or 16k respectively. The goal of both task is to evaluate the model’s ability to accurately identify unique relevant information from a large volume of irrelevant documents (or KV pairs).
Real-World Long Context Tasks. The current mainstream real-world long context tasks are derived from LongBench (Bai et al., 2023), primarily including single-document question answering, multi-document question answering, summarization, code completion, with the average length of most tasks ranging from 5k to 15k. We use the metrics and scripts provided along with the benchmark for evaluation.

heading: 4.3 Data Selection
text: In the experiments, we truncate the length M of all data to 32768 and set the segment length L to 128, thereby dividing each data instance into N = 256 segments. We set the hyper-parameters α = β = 1 in LDS. To improve ProLong’s computational efficiency, we replace the standard LDS with LDSsp, utilizing a sampling size of T = 5000. We adopt a small model OPT-350m (Zhang et al., 2022) for calculating perplexity in LDS. We rank the book and code data based on their LDS, independently retaining those documents with high LDS from each source. These retained subsets, plus the pre-training dataset, are then integrated to construct our final training dataset. This method not only ensures the quality of our data but also enhances its diversity. In the experiments, we primarily compare the results of utilizing the entire dataset (Full), randomly selecting a 50% subset (Rand) and selecting the top-scoring 50% subset (ProLong) for extending training. We list the average LDS statistics under different data selection strategies in Appendix A.2.

heading: 4.4 Training Details
text: We extend the pre-trained Llama2-7b/13b (Touvron et al., 2023b) models to support context windows as large as 32768 tokens. We employ the NTKaware (bloc97, 2023) method with base 160000 to rescale the position indices without any additional modifications to the Llama2 model architectures. Other hyper-parameters for training are displayed in Appendix B.1. For model comparison, we evaluate several mainstream LLMs with long context capability listed in Appendix B.2.

heading: 5 Experimental Results

heading: 5.1 ProLong Effectiveness
text: As shown in Table 1, we compare the long context performance of the models on different data selection methods (i.e., Full v.s. Rand 50% v.s. ProLong
50%). The evaluated benchmarks include KV retrieval, multi-document QA, and typical tasks in LongBench. We find that based on the same model (Llama2-7b or Llama2-13b), ProLong consistently outperforms random data selection across all the long context benchmarks. This result underscores the efficacy of ProLong in mining data with strong semantic dependencies across long contexts, which in turn enhances the long-context capabilities of LLMs. Interestingly, Table 1 also suggests that training with 50% of the data selected by ProLong often yields better results than training with the full dataset. This observation implies that most long contexts in the full dataset are not truly long at all. Surprisingly, the principle of “less is more” applies to certain specific long-context tasks.

heading: 5.2 Ablation Study on ProLong
text: As shown in Table 2, we also conduct ablation studies on multi-document question answering tasks. We first investigate the effect of the dependency specificity by removing it (denoted as “w/o DSP”), which is motivated to mitigate trivial dependencies
introduced by repetitive patterns. The accuracy decreases in two different total document settings. Moreover, we observe a significant presence of repetitive patterns in the top-ranked data under the “w/o DSP” setting, with examples provided in Appendix E. Thus, the dependency specificity term indeed plays a crucial role in preventing the retrieval of meaningless repetitive patterns. Next, we examine the effect of using DSP as a multiplier instead of an addend in Eq. 6 (refer to Appendix D for the formula with DSP as an added bias, denoted as “w/ DSP-add”). The results indicate that using DSP as a bias is insufficient to mitigate the trivial dependencies introduced by repetitive patterns, and the performance of experiments is similar to that without DSP. Finally, we conduct experiments to analyze the impact of removing the dependency distance (denoted as “w/o DDI”) or removing the dependency strength (denoted as “w/o DST”) to do an ablation study. The results show that depen-
dency strength, as the core aspect, contributes the most to overall performance. Furthermore, removing DDI also leads to a decrease in performance. In conclusion, each term in the LDS calculation has its unique advantages and is indispensable in our ProLong framework. DST, DDI, and DSP collectively make the final LDS a feasible and effective index for mining data with long dependencies amidst a sea of long texts.
Given the effectiveness of ProLong, we train models from Llama2 (Touvron et al., 2023b) on the top of ProLong 50% (refer to Sec. 4.3 for setups in detail), resulting in ProLong-7b/13b.

heading: 5.3 Performance on Language Modeling
text: In Table 3, we compare our models ProLong7b/13b with other baseline models in language modeling in various context window size, which is measured by perplexity. Firstly, ProLong-7b/13b exhibit a trend where perplexity decreases as the context window size increases, suggesting that they are capable of performing better language mod-
eling with longer contexts. Additionally, under any context window size, our models based on ProLong significantly outperform other baseline models, including Llama2. This demonstrates that ProLong-7b/13b exhibits superior language modeling capabilities in long contexts.

heading: 5.4 Performance on Key Value Retrieval
text: We claim that lower PPL does not fully encapsulate the ability to handle long text tasks effectively. Thus, we also present the key value retrieval performance of ProLong-7b and ProLong-13b in Table 4. Compared with other models based on Llama2, ProLong-7b/13b surpasses them by a significant margin under the settings of 140 and 300 key-value pairs. Notably, ProLong-13b even outperforms the commercial model GPT-4-32k, especially in the more challenging setting of 300 key-value pairs. One key gap lies in that in the middle position, while other models show a significant drop in performance, our models based on the ProLong framework maintain high performance. This suggests
that the “lost in the middle” phenomenon, as described by Liu et al. (2023), is largely alleviated in our models. We also conducted experiments on the “needle in a haystack” task, which bears similarity to the key-value retrieval task. Detailed experimental results can be found in Appendix C.1. Consistent with the key-value retrieval task, the results demonstrate that the long context retrieval capability of ProLong-7b/13b experiences almost no degradation in the middle part of the context.

heading: 5.5 Performance on LongBench
text: For comprehensive evaluation on the long context performance, we further use real-world long context benchmarks (Bai et al., 2023). As shown in Table 5, we report the performance results (%) of partial English and code tasks on the LongBench dataset. Specifically, our ProLong-7b/13b achieves 41.1%/44.4% overall accuracy, which outperforms all the compared Llama2-based models by large margins. Besides, ProLong-13b surpasses the commercial model GPT-3.5-turbo-16k by an absolute gap of 2.4%. These results not only confirm the robust long-context capabilities of ProLong-7b/13b but also suggest that the ProLong framework holds promise for effectively addressing real-world longcontext tasks. We also conducted experiments on standard short-length benchmarks, which further demonstrates ProLong can enhance the longcontext modeling ability without compromising other capabilities. Detailed experimental results can be found in Appendix C.2.

heading: 6 Analysis

heading: 6.1 Balance between Accuracy and Efficiency
text: Initial LDS requires N2 times precise calculations. In practice, sampling T ≪ N2 times and using a small model for perplexity calculations provides a more efficient, albeit approximate, solution. In this section, we explore the balance between accu-
racy and efficiency. We conduct experiments on the impact of different model sizes used for calculating perplexity (PPL) and different sampling granularity mentioned in Sec. 3.5. To assess the accuracy of ProLong’s capability in selecting longdependency data, We construct a toy test set from various sources. Specifically, we manually construct 100 instances with strong long-dependency as positive examples and 100 instances with weak long-dependency as negative examples. Subsequently, we use ProLong to score and rank the test set, assessing the number (accuracy) of positive examples contained in the top 100 ranked data.
We calculate the accuracy of ProLong in the test set to retrieve long-context documents and the number of documents processed per second as a measure of speed. The experimental results are as shown in Table 6, which reveals that smaller models and sampling sizes can achieve superior speed while sustaining comparable accuracy. In our experiments, we empirically choose OPT-350m and 5000 sampling size to perform ProLong, which may not necessarily yield the best results. Practically, it is essential to select an appropriate model and sampling size to strike a balance between accuracy and efficiency.

heading: 6.2 Visualization Analysis
text: To better illustrate the superiority of ProLong, we provide visualizations of dependency strength heat
maps for various documents under the same settings. Figure 3 displays various documents from the training set alongside their heat map visualization and LDS. LDS accurately captures the long-range dependencies within diverse documents. Specifically, academic papers typically showcase high LDS values, indicative of a more intricate heat map relationship and a pronounced presence of long-range dependencies. The same goes for and vice versa. Besides, we also find that the results of random concatenation of short texts (Figure 3.c) are very similar to those in storybooks (Figure 3.b), which demonstrates that even in naturally long texts, long-range dependencies may be minimal. Moreover, structured concatenated documents receive much higher LDS scores than randomly concatenated documents, corroborating part of the conclusions from the concurrent work, SPLICE (Staniszewski et al., 2023).

heading: 7 Conclusion
text: In this study, we propose ProLong, a novel and efficient approach for filtering long-dependency data in language model extension training. Benefiting from ProLong, we demonstrate that longdependency data is the key to enhancing longcontext modeling capability. Moreover, low-quality short-dependency data may even impair longcontext modeling capability. Extensive experiments on multiple long-context benchmark datasets demonstrate the effectiveness of our ProLong in enhancing the long-context modeling capability of large language models. We hope ProLong can inspire researchers to spend more effort on data of more intrinsic long-dependency rather than long surface form.
Limitations
The ProLong framework can be used to identify documents that carry long dependencies effectively, and LLM trained on these documents exhibit significantly enhanced long-context modeling capabilities. However, to build strong LLMs, ProLong should be used in combination with other techniques. Here we list some of the limitations that are not considered when designing ProLong: (1) Diversity of Corpora. Our experiment only considers English books and code texts. In future work, we should extend our analysis to cover a wider range of long-context data, including different languages and document types. (2) Model Consider-
ations. In this study, our experiments are solely performed on Llama2-7B and Llama2-13B models. It is expected to obtain better performance with larger base models. (3) Data Mixture Percentage. We do not perform comprehensive exploration of the data mixture percentage, which is reported to be important to the final performance of LLM.
In future studies, we hope to explore the abovelisted limitations further.
Ethics Statement
Our study does not carry any ethical concerns. Specifically, our training data are publicly available and designated for research purposes only. We inspect our dataset to ensure it does not contain any unethical content, private information and offensive topics. Moreover, the base models we used are also publicly available for research purposes.

heading: Acknowledgments
text: Min Yang was supported by National Key Research and Development Program of China (2022YFF0902100), National Natural Science Foundation of China (62376262), the Natural Science Foundation of Guangdong Province of China (2024A1515030166), Shenzhen Science and Technology Innovation Program (KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039).

heading: A Training Datasets
text: A.1 Training Datasets Overview
All training corpus we used in our experiments are given in Table 7. The majority of the data consists of English books and code, all of which have lengths exceeding 32k.
A.2 Average Long-Dependency Score
In the experiments, we primarily compare the results of utilizing the entire dataset (Full), randomly selecting a 50% subset (Rand), and selecting the top-scoring 50% subset (ProLong) for extending training. We analyze the average long-dependency scores in these three scenarios, as detailed in Table 8, where the results for Rand and Full are very close across different data sources, but the results for ProLong are significantly higher than both of them.

heading: B Experimental Details
text: B.1 Training Hyper-parameters
All model variants are trained via the next token prediction objective. We set a learning rate of 2× 10−5 with no weight decay and the whole training step is set to 6000 with a global batch size of 128. For subsequent experiments with 50% data setting, the training step is set to 3000 steps. Additionally, we adopt a linear learning rate warm-up scheduler over 20 steps and the AdamW (Loshchilov and Hutter, 2017) optimizer with β1 = 0.9 and β2 = 0.95.
B.2 Model Baselines
We evaluate several mainstream LLMs with long context capability, including GPT-4-32k (Achiam et al., 2023), GPT-3.5-Turbo-16k (OpenAI, 2022), Llama2-7b (Touvron et al., 2023b), Llama-13b, Code Llama-7b (Roziere et al., 2023), Yarn-7b-64k (Peng et al., 2023), Yarn-13b-64k, Llama2-7B-chat4k (Touvron et al., 2023b), Llama2-13B-chat-4k, LongChat-v1.5-7B-32k (Li et al., 2023a), Vicunav1.5-7B-16k (Chiang et al., 2023), Vicuna-v1.513B-16k, Longlora-7B-16k (Chen et al., 2023b), and PI-Llama2-13B-16k (Chen et al., 2023a).

heading: C More Experimental Results
text: C.1 Needle in A HayStack Task A simple ‘needle in a haystack’ analysis to test in-context retrieval ability of long context LLMs. This test involves placing a random fact or statement (the ‘needle’) in the middle of a long context window (the ‘haystack’), requiring the model to retrieve this needle. Then, it iterates over various document depths (where the needle is placed) and context lengths to measure performance. In the setting where the maximum context length is 32k, the results in Figure 4 show ProLong-13B obtains 100% needle in a haystack accuracy across all tested depths and context lengths. ProLong-7B exhibits a few errors when the context length exceeded 16k, it still maintains an overall accuracy of 95%.
C.2 Standard short-length benchmarks As shown in the table 9, we evaluate ProLong7b and ProLong-13b on several standard shortlength benchmarks from Hugging Face Open LLM Leaderboard (Beeching et al., 2023), i.e., 25-shot ARC-Challenge (Clark et al., 2018), 10- shot Hellaswag (Zellers et al., 2019), 5-shot MMLU (Hendrycks et al., 2020), 0-shot TruthfulQA (Lin et al., 2021), 5-shot Winogrande (Sakaguchi et al., 2021), 5-shot GSM8K (Cobbe et al., 2021). When compared with the baseline Llama27b and Llama2-13b models, we find that ProLong does not exhibit a significant performance decline,
on the contrary, it achieves comparable or even better performance than baseline.

heading: D Additive Dependency Specificity
text: Additive Dependency Specificity is calculated as follows:
LDS = N∑
i=1
i−1∑
j=1
{ ( αDSTi,j + βDDIi,j + γDSPi )× Ii,j } (9)
where γ is a hyper-parameters. In the ablation experiments, we set the hyper-parameters α = β = γ = 1 in LDS.

heading: E Repetitive Patterns
text: As shown in Figure 5, we present some examples of repetitive patterns in the training data. To be specific, the repetitive patterns in C++ are a huge number of reptitive numbers, in the arXiv are numerous repetitive formula symbols, and in the PG19 are an abundance of reptitive tables, which may potentially have a negative impact. When we do not use dependency specificity or use additive dependency specificity, this type of data will receive a high long-dependency score, thereby impacting the quality of the filtered data.

heading: Future Work
text: In future work, we should extend our analysis to cover a wider range of long-context data, including different languages and document types. (2) Model Consider-
ations. In this study, our experiments are solely performed on Llama2-7B and Llama2-13B models. It is expected to obtain better performance with larger base models. (3) Data Mixture Percentage. We do not perform comprehensive exploration of the data mixture percentage, which is reported to be important to the final performance of LLM. In future studies, we hope to explore the abovelisted limitations further