Scaling Long Context Training Data by Long-Distance Referrals

Authors: Yonghao Zhuang, Lanxiang Hu, Longfei Yun, Souvik Kundu, Zhengzhong Liu, Eric P. Xing, Hao Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: long context training; continue training; dataset;

Abstract:
Training large language models for long context understanding faces the challenge of data shortage.
Previous data engineering approaches mechanically concatenate short documents, which may create many pseudo long documents but raise concerns about data quality.
In this paper, we study the core attribute of high quality data for long context training, and provide a data pipeline, LongPack, to scale
such data.
We found that long distance referrals, which occur in natural long documents, are crucial for long-context training.
However, simply concatenating short documents does not reliably generate these relations.
We further show that the density of long-distance referrals, which is higher in longer documents, has a key role in training efficiency, making previous upsampling methods suboptimal.
To enrich long documents, we propose LongPack, a data pipeline that constructs long documents by packing shorter ones based on referral relationships.
Specifically, for web pages, which are the primary source for language model training, we found hyper-link a native signal for such a relation.
By packing web pages through their hyper-link connection, we can create longer, high-quality documents.
Our experiments demonstrate that LongPackis highly scalable, generating a corpus of long documents equivalent in size to an entire pretraining dataset using just 0.5% root documents.
Furthermore, the constructed documents have a ‘near-natural’ quality as innate long documents for long context training, reaching a 32.7% higher score than previous state-of-the-art methods.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729065469740

thread_type:
official_review

<thread_object>
creator: reviewer_kwvs

<content>
summary:
The paper introduces LongPack, a novel data pipeline designed to address the scarcity of high-quality long documents for training LLMs with long-context understanding capabilities. The authors argue that long-distance referrals are crucial for effective long-context training, and design controlled experiments to prove this point. The method, LongPack, uses the hyperlinks in web data to create long distance referrals, which is low-cost and practical. And datasets curated by this method can more greatly improve long-context abilities of LLMs.

strengths:
1. The importance of long distance referrals is well proved by enough experiments, including both adding and reducing them.
2. The method, LongPack, is easy to understand and practical. It only needs simple steps, but can construct many training samples. This would be very helpful in solving the data scarcity issue in long-context post-training stage of open-source LLMs.
3. The training datasets are sufficiently large, and the samples' text length is up to 128k. The investment on the experiment is great.

weaknesses:
1. Format errors

There are many errors in word separation, capitalization, and citation format, such as "LongPackis highly" (line 25), "textrank Nathan (2016)" (line 323), "basic niah test" (lines 382), etc.

2. Messy experiment results

The positions of the charts and the organization of sections are somewhat casual, making the main results scattered, which may reduce readability. For example, there are 4 bar charts scattered in 4 different pages, with each of them contains some experiment results. The author should use a individual section to gather the main results of the experiments. 

3. Lack of baselines

There is a lack of baselines when evaluating the efficacy of LongPack (Table 1 and 2). The author only adopts a naive baseline using length up-sampling. But as far as I know, there are many other approaches for constructing or selecting higher-quality long-context training data, such as ProLong [1]. Moreover, training with more base models such as Llama would be better.

[1]Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models

questions:
Can you include more related works about constructing or selecting higher-quality long-context training data as baselines? I know long-context training is very costly, so you can use a dataset of a smaller size.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540293136

thread_type:
official_review

<thread_object>
creator: reviewer_qbkr

<content>
summary:
- This paper proposes a data engineering pipeline to generate long documents for large language model training based on long-distance referrals.

strengths:
- Constructing high quality pre-training datasets is a very important topic in LLM research.
- The authors' idea is an automated build methodology inspired by real-world industrial experience.
- The authors' experiments are large and relevant on datasets of a certain scale.

weaknesses:
- Typos: LongPackis --> LongPack is (Line 25)
- The content of the paper is more oriented towards practical applications and may lack a technical contribution to some extent.
- The formatting of the references is inconsistent, in some places the references have parentheses, in others they don't, which should be caused by the use of \cite in the latex source code.
- The pipeline can be better described in this paper by pseudocode rather than natural language descriptions
- The experimental section of the main text shows less content, so consider adding some of the content from the Appendix to the experimental section of the main text.

questions:
- Can the authors explain the choice of the LLM backbone, only the GLM model of 9B is currently used in the experiments. Why don't you consider using models from the more current mainstream LLM series in your experiments? **I realize that it is more difficult to add experiments on top of such a task, so the authors do not need to add new experimental data, but need to explain why.** As far as I understand, GLM should be a bit stronger than llama in terms of Chinese language capability, but are the Chinese language data and Chinese language tasks included in the data and tasks used for the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672824275

thread_type:
official_review

<thread_object>
creator: reviewer_7fcd

<content>
summary:
This paper aims to tackle the scarcity of high-quality, lengthy documents in training datasets, which is a critical challenge in the training of LLMs for long-context understanding: The authors propose a new data engineering pipeline, LongPack, which aims to construct long documents by leveraging referral relationships, particularly hyperlinked structures in web pages, to mimic the naturally occurring long-distance referrals found in genuinely long documents. This approach addresses both the need for extensive context length in training data and the quality issues inherent in previous methods that simply concatenate short documents. Experiments on the RULER benchmark, which contains 13 tasks, demonstrate the effectiveness of LongPack in comparison with the GLM-4 baseline and multiple heuristics for creating lengthy training samples.

strengths:
+ This goal of constructing long-distance referrals is well-motivated. Using semantically related tokens separated by a significant distance in pre-training will make the LLMs more capable of dealing with long contexts.

+ The use of hyperlinks in data packing is simple and intuitive. The authors demonstrate the benefit of utilizing such natural signals of referral relationships between documents, especially on the web.

+ The proposed framework is efficient, producing a corpus of long documents equivalent to an entire pretraining dataset using only 0.5% of the original documents.

weaknesses:
- The idea of using links between documents during language model pre-training has been explored in [1], where text segments from two linked documents are concatenated together as input. I feel the packing idea proposed in this paper bears similarity to that in [1]. Therefore, the technical novelty is somehow limited.

- Statistical significance tests are not conducted. It is not clear whether the gaps between LongPack and the baselines are statistically significant or not in Tables 2 and 7. In fact, the improvement is subtle in some columns. 

- The study focuses heavily on web-based documents. It would be valuable to explore how long-distance referrals could be identified and leveraged in other domains, such as academic papers.

[1] LinkBERT: Pretraining Language Models with Document Links. ACL 2022.

questions:
- Could you conduct a statistical significance test (e.g., two-tailed t-test) to compare LongPack with the baselines in Tables 2 and 7, and report the p-values?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712744413

thread_type:
official_review

<thread_object>
creator: reviewer_i48t

<content>
summary:
This paper introduces a new long-context training dataset generate pipeline called LongPack. Authors emphasize the importance of (super) long distance referrals, pairs of tokens that are semantically the same but has a long distance in a document. LongPack utilizes the hyperlinks from the raw HTML to construct long texts with long distance referrals. With packed data generated by LongPack, long-context training improves 32.7% compared to previous data generation recipe.

strengths:
1. Studying workflow of generating datasets for long-context training is interesting and of importance for the extending the length of context window of large language models. 
2. This paper emphasis the importance of long distance referrals to high quality long-context training datasets. It introduces an intuitive and effective pipeline LongPack to contract such high quality datasets. 
3. The empirical experiments shows the effectiveness of the packed data constructed by LongPack, compared to simple upsampling strategy.

weaknesses:
1. The paper is not well-ordered. For example: 
    (1). Section 3.1 present the experiment results first, which refers to Figure 3 that is located a few pages after the text. The experiment section also refers Figures in a few pages ahead, making the reading experience not fluent.  
    (2). Some of the content from Section 4.3 are already introduced in Section 3.2. 
2. Some details are missing(see question 1 - 3)
3. Some of the result analysis are not sufficient and well supported (see question 4-5)
4. In Table1, the simple “Upsample” strategy is compared to LongPack. Considering there are some long-context training dataset, for example books and long dialogues introduced in Section 2. 
5. Minor points:
    (1). In 025 line: LongPackis highly scalable —> LongPack is highly scalable
    (2). In 076 line: propose LongPackto solve the shortage… —> propose LongPack to solve the shortage… (similar errors occur also in other places)
    (3). $d_0$’s  meaning is not explained. I would assume it is the distance threshold. 
    (4). The citation format in Section 4 Experiment is not in standard.  
    (5). In line 483-484, it is said “we report the validation loss before and after training with our data. The result is shown in Table 2”. The caption of Table 2 is “The general performance before and after training with our data”. “the general performance” does not match the “validation loss”, making me confused about the numbers in the Table 2.

questions:
1. How are the scores of dataset examples calculated in Figure 1(a)?
2. How the length of the document is calculated, by token, by word or by sentence? 
3. How are the order of the retrieved contents decided?
4. Why the #referral in the referral distance bucket of 512-2048 from docs with length (16k, 32k) in the packed documents significant drop compared to the refinedweb dataset?
5. It is said that in Figure 5 (b), the performance of (16k-64k) group is close to (15k-32k) group. This statement is not accurate, for example, in multi-pos retrieval, based on (16k-64k) group, the performance is significantly better. While in multi-hop tracing, based on (16k-64k) group, the performance is significantly worse. Thus the conclusion “This indicates that the more long document it use, the better performance it obtains” is not well-supported by the experiment results. 
6. How is “near-natural” property of the generated long documents been evaluated? 
7. Any other ways of contracting packed document are explored, apart from packing the pages by prepending all retrieved contents before the root document?
8. What is the performance of the model trained on existing long-context training dataset (such as using books or long dialogues) compared to the datasets generated by LongPack?
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings

This paper introduces LongPack, a data pipeline that generates high-quality long-context training data for large language models (LLMs). The authors argue that traditional methods of concatenating short documents fail to capture long-distance referrals, a key semantic feature of natural long documents. LongPack uses hyperlinks from web pages to maintain semantic linkages, creating more informative training data.

Key findings:
1. LongPack achieves higher referral density than traditional methods, ensuring meaningful semantic connections.
2. The method is highly scalable, requiring only 0.5% of the original documents to generate datasets of comparable size.
3. Models trained on LongPack datasets show a 32.7% improvement in long-context tasks.
4. The generated data is of near-natural quality, as validated by benchmarks like RULER.

(b) Strengths of the Paper

1. Innovative Approach: The hyperlink-based document construction addresses the shortcomings of concatenation methods, preserving important semantic relationships.
2. Scalability: LongPack generates large datasets efficiently, offering a resource-effective solution for long-context training.
3. Improved Performance: Empirical results show a significant boost in model performance, with a 32.7% improvement in relevant tasks.
4. Data Quality: The generated documents exhibit high referral density and near-natural quality, suitable for robust LLM training.

(c) Weaknesses and Missing Elements

1. Organization and Clarity: The paper could be better structured, with definitions and methodologies presented before results to improve readability.
2. Analysis Gaps: The paper lacks sufficient explanation of referral density variances across different document lengths, which would strengthen the findings.
3. Limited Comparison: The paper doesn’t compare LongPack with other referral-based strategies or naturally long datasets, such as books, which could provide a broader context.
4. Domain and Model Limitations: The focus on web pages may limit the generalizability of the results. Additionally, the choice of the GLM-4 model may restrict the applicability to other architectures.

(d) Reasons for Acceptance/Rejection

While LongPack offers an innovative, scalable solution with significant improvements in model performance, the paper would benefit from better organization, deeper analysis of results, and a broader comparison with other methods.