Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

## Abstract

Abstract Large language models have demonstrated remarkable potential in various tasks, however, there remains a significant scarcity of open-source models and data for specific domains. Previous works have primarily focused on manually specifying resources and collecting high-quality data on specific domains, which significantly consume time and effort. To address this limitation, we propose an efficient data collection method Query of CC based on large language models. This method bootstraps seed information through a large language model and retrieves related data from public corpora. It not only collects knowledge-related data for specific domains but unearths the data with potential reasoning procedures. Through the application of this method, we have curated a high-quality dataset called Knowledge Pile , encompassing four major domains, including stem and humanities sciences, among others. Experimental results demonstrate that Knowledge Pile significantly improves the performance of large language models in mathematical and knowledge-related reasoning ability tests. To facilitate academic sharing, we open-source our dataset and code, providing valuable support to the academic community.

## 1 Introduction

Large language models (LLMs) are becoming the new trend not only in natural language processing but also in the entire AI community, pioneered by OpenAI ChatGPT and GPT-4 OpenAI ().
While commercial LLMs are close-sourced, open-source models such as LLaMA Touvron et al. () and Mistral Jiang et al. () are widely studied by the community since they serve as general base models for building LLM applications.
Based on these base models, domain-specific models, show great potential in specific domains, such as medicine Yang et al. (); Gao et al. (), finance Wu et al. (); Zhang and Yang (), science Taylor et al. (); Wei et al. (), and law Nguyen (); Cui et al. ().
These domain-specific enhanced models are based on specific human-crafted data recipes Azerbayev et al. (); Wang et al. ().

Figure: (a) Manual Data Collection
Refer to caption: x1.png

However, crafting domain-specific data is very costly. As depicted in Figure 0(a), traditional data collection methods involve the selection of relevant resources by domain experts, followed by data collection and processing by engineers. On the one hand, such endeavors are highly labor-intensive, requiring several months of collaboration between multiple domain experts and engineers for corpus collection. On the other hand, some specific domain-related data distribution may be highly scattered, which poses many challenges for large-scale domain-specific data collection.
Therefore, in this paper, we introduce an automatic strategy to retrieve data from public corpora for specific domain knowledge bootstrapping, which we call Query of CC.

In Query of CC, we initially collected seed information in some specific domains, such as keywords, frequently asked questions, and textbooks, to serve as inputs for the Query Bootstrapping stage. Leveraging the great generalization capability of large language models, we can effortlessly expand the initial seed information and extend it to an amount of domain-relevant queries. Inspiration from Wang et al. () and Xu et al. (), we encompassed two stages of expansion, namely Question Extension and Thought Generation, which respectively extend the queries in terms of breadth and depth, for retrieving the domain-related data with a broader scope and deeper thought. Subsequently, based on the queries, we retrieved relevant documents from public corpora, and after performing operations such as duplicate data removal and filtering, we formed the final training dataset.

Otherwise, leveraging Query of CC, we collect a high-quality knowledge dataset Knowledge Pile, which starts from some seed information of four major domains, including STEM, humanities, social sciences, and medical sciences, as well as general knowledge.
Utilizing Knowledge Pile, we enhance the Llama and Mistral models through continuing learning. Experimental results indicate that through the Knowledge Pile, both Llama and Mistral enhanced models achieved significant performance improvements over baselines in benchmark tests related to mathematics, knowledge assessments, professional examinations, and some complex reasoning tasks.

To sum up our contribution:

- •
We propose Query of CC, a self-bootstrapping strategy for automatic data collection to enhance domain-specific knowledge from public corpora.
- •
We collect and release Knowledge Pile, based on various categories such as STEM, human science, and social science. The dataset effectively enhances the model’s abilities in mathematical and knowledge-based reasoning.
- •
We introduce Llama-QoC and Mistral-QoC models trained on Knowledge Pile, which demonstrate significant improvements on several professional exams and reasoning datasets.

## 2 Related work

### 2.1 Large language model for knowledge-based reasoning

In recent years, significant progress has been made in the field of Natural Language Processing (NLP), driven by the emergence of large language models OpenAI (); InternLM-Team (); Bai et al. (); Sun et al. (). Particularly, in the domain of academic and professional examinations, Some language models such as ChatGPT and GPT-4 OpenAI () have demonstrated remarkable success in solving complex tasks, achieving human-like performance through the utilization of the capability of reasoning Wei et al. (); Wang et al. (). However, open-source language models lag behind in performance (Like Llama Touvron et al. (), Mistral Jiang et al. () etc.), possibly due to a lack of data.

### 2.2 Manual Data Collection

Currently, extensive efforts are being dedicated to the manual collection of specific training data to enhance the capabilities of Large Language Models (LLM) in knowledge-based reasoning. In the field of mathematics, Lewkowycz et al. () undertook the task of gathering approximately 40 billion tokens of data from arXiv and web math pages. They developed a series of Minerva models based on PaLM Chowdhery et al. () and observed that augmenting the model with more mathematical data significantly enhances its proficiency in mathematical reasoning. Similarly, numerous works Azerbayev et al. (); Wang et al. (); Paster et al. () have undertaken the collection of mathematics-related data, encompassing papers, web pages, and code, with considerable cost.

In the academic and technological domains, Taylor et al. () collecting 106 billion academic and technological data. They asserted that the resulting 120B Galactica model surpasses GPT-3 in various academic benchmarks. These works highlight the efficacy of manual data collection in enhancing model performance. However, it is crucial to note that these data collection endeavors are labor-intensive, lack diversity, and present challenges in scalability, thereby posing some constraints on the overall improvement of model performance.

Figure: Figure 2: The overview of Query of CC’s two major components: Query Bootstrapping and Data Retrieval.
Refer to caption: x3.png

In contrast to these human-centric collection methods, our approach delves into an automated method for bootstrapping and collecting domain-specific data from public corpora and aims to achieve scalability, efficiency, and cost-effectiveness in data collection.

### 2.3 Retrieval-based Data Collection

For large language models, many works Li et al. (); Li and Qiu () are based on retrieval to enhance their capabilities. The majority of these focus on retrieving documents relevant to the questions to improve the model’s prior knowledge, thereby enhancing the performance on knowledge-related tasks and reducing hallucinations. Some works Yao et al. () attempt to use retrieval during the training phase for data selection to improve specified downstream tasks. However, the data selected by such methods lacks diversity on the one hand, with the selection scope heavily dependent on the choice of downstream tasks, and on the other hand, it is similar to manually collected data, unable to be expanded in an automated manner. In comparison to these retrieval-based methods, our approach utilizes the bootstrapping method of large models, resulting in more diverse queries for data retrieval, making the training data more diverse and scalable.

## 3 Query of CC

### 3.1 Overview

The overview of Query of CC is illustrated in Figure 2. This framework encompasses two major stages, namely Query Bootstrapping and Data Retrieval. During the Query Bootstrapping phase, the Large Language Model constructs questions and answers centered around the given keywords to retrieve relevant text from public corpora. Through this bootstrapping process, we construct queries around the seed keywords in both depth and breadth, ensuring that the query encompasses a wide range of knowledge and sufficient depth. In the Data Retrieval phase, we employed the BM25 algorithm to retrieve text relevant to specific categories and obtained training data through post-processing steps such as deduplication.

### 3.2 Query Bootstrapping

To efficiently and comprehensively retrieve high-quality data relevant to the given seed information, we employed the Query Bootstrapping method, inspiration from Wang et al. (). During this phase, we expanded queries around seed information such as category or some related questions, by querying large language models and aims to extend the scope of retrieved data from both breadth and depth perspectives.

#### Question Extension

To broaden the scope of our queries, we leverage large language models to generate questions relevant to the given seed words which we called Question Extension. Leveraging the capabilities of large models for question generation is evident in expanding the conceptual boundaries of the keywords. This process evolves a narrow category into a more comprehensive representation, encompassing various concepts associated with the given words. Consequently, it significantly enhances the breadth of the query set, ensuring a more comprehensive coverage of various aspects within the target domain. Additionally, to ensure diversity in generation, we set different seeds for LLM inference during experimentation, and employ a higher temperature during the sampling process, thereby ensuring the diversity of the generated questions.

#### Thought Generation

In addition to expanding the range of queries through Question Evolution, we are dedicated to enhancing the depth of thought in generated queries through Thought Generation. In this phase, we are inspired by the work of Wei et al. (), employing large-scale language models to produce the cognitive processes necessary for answering questions. This enables us to acquire detailed and insightful responses. This approach supports a more thorough exploration of concepts related to seed information and captures the cognitive processes essential for answering questions. The generated thoughts not only serve for retrieving seed information-related data but also, by expanding queries, offer additional context and relevant knowledge, contributing to a more comprehensive understanding of the subject matter.

#### Query post processing

We conducted post-processing operations on queries, involving two main stages: pre-cleaning before entering the seed information pool and cleaning and filtering before entering the query pool. On one hand, the generated questions underwent a re-cleaning process and were incorporated into the seed information pool for the next rounds of question bootstrapping and thought generation. This cleaning step involved removing incompletely generated language data to prevent the influence of non-natural language. On the other hand, before entering the query pool, we employed Minhash-LSH deduplication on queries to mitigate performance inefficiencies resulting from redundant retrievals.

### 3.3 Data Retrieval

Based on the query bootstrapping stage, we get extensive and in-depth queries. During the data retrieval stage, utilizing the enriched queries, we employ the BM25 Robertson and Walker () algorithm to retrieve data from general public corpora. BM25 is a widely adopted relevance calculation method commonly used by search engines. It calculates the relevance score between the given query and target documents by weighting and summing the matching degree of keywords in the query with the target documents. Efficiency is the reason why we use BM25 to calculate the relevance. When dealing with billion data, performing relevance calculations for each query against every document becomes exceedingly challenging, while BM25 rapidly retrieves documents relevant to the target query. Compared with Dense Retriever Karpukhin et al. (), it may incur a potential loss in retrieval accuracy, but the latter comes with an unbearable high computational cost. Exploring the potential impact of retriever selection on the quality of collected data might be a pivotal direction for future research.

For each query $q_{i}$, we conduct the relevance score against every document $d$ in the public corpora $\mathcal{D}$. Followed by sorting the documents based on relevance, we retrieve top-k document set $\mathcal{S}_{i}=\{\widetilde{d}_{1},...,\widetilde{d}_{k}\}$ with the highest relevance with the query $q_{i}$. In our experiments, the typical choice for $k$ is 1000. The retrieved data which related all the query $q_{i}\in\mathcal{Q}$ is consolidated into training dataset $\mathcal{S}=\cup_{i}\mathcal{S}_{i}$.

### 3.4 Data Post-processing

To enhance the quality of the retrieved dataset, we conducted a series of filtering procedures. Taking inspiration from the filtering methods proposed by  Wenzek et al. (), we trained a scoring model to detect high-quality data. This model was trained by the selected high-quality data as positive samples and using the randomly sampled instances from Common Crawl as negative examples. we used this model to score all the retrieved data, and a specified threshold was applied to filter the data. Also, we filter the non-English documents and the code in retrieved data.

Additionally, considering the detrimental impact of data duplication on model training, we employed the Minhash-LSH and n-gram methods to deduplicate the retrieved data. Duplicate data with a similarity threshold of 0.8 was removed from the dataset.

**Table 1: Comparation of Knowledge Pilewith other specific domain knowledge dataset. In this table, most data scales are derived from publicly released research papers, while the data scale for the Knowledge Pile is obtained through tokenized data analysis using the Llama2 tokenizer.**
| Datasets | Target domain | automatic | methods | source | Data scale | Open Source |
| --- | --- | --- | --- | --- | --- | --- |
| Proof of Pile | Mathematics reasoning | ✗ | human collection | arXiv, Textbooks, Lib., Stack Exchange, ProofWiki, MATH | 8.3B | ✓ |
| OpenWebMath | Mathematics reasoning | ✗ | human collection | Common Crawl | 14.7B | ✓ |
| MATHPILE | Mathematics reasoning | ✗ | human collection | arXiv, Textbooks, Lib., Stack Exchange, ProofWiki, MATH,Web | 9.5B | ✓ |
| AcademicGPT | Academic | ✗ | human collection | arXiv, Unpaywall,Top Universities, Pubmed, Common Crawl, Semantic Scholar, Wiki | 370B | ✗ |
| Galactica | Academic | ✗ | human collection | Papers, Code, Reference Material, Knowledge Bases, Common Crawl, Prompts | 106B | ✗ |
| Knowledge Pile (ours) | Mathematics reasoning & Knowledge | ✓ | automatic query and retrieve | Public Corpora | 188B | ✓ |

## 4 Knowledge Pile

Leveraging Query of CC, Based on queries of some knowledge categories, we retrieved high-quality knowledge data from processed public corpora. We call the collected datasets as Knowledge Pile. In this section, we will introduce the analysis of queries (Section 4.1) and the statistic information of Knowledge Pile (Section 4.2), See more implement details in Appendix C.

### 4.1 Query Analysis

Figure: Figure 3: The category distribution of the query for Knowledge Pile.
Refer to caption: x4.png

The progress of query bootstrapping initiates from some categories. Inspiration of the classification of Hendrycks et al. (), we select multiple categories for our initial seed information in the STEM (Science, technology, engineering, and mathematics), Humanities sciences, Social Sciences, and miscellaneous. The key keywords for each category are as follows:

- STEM: mathematics, physics, chemistry, biology, computer science, engine;
- Humanities: logical, history, law, philosophy, religions;
- Social science: econometrics, politics, psychology, sexuality, public relations, psychology, sociology;
- Misc: medicine, virology, commonsense knowledge and other miscellaneous.

After multiple rounds of iterative augmentation and deduplication, we obtain a total of 340,000 queries. The distribution of queries across different domains is depicted in Figure 3. In our query pool, STEM-related queries constitute the majority, while the proportion of queries similar to miscellaneous is relatively small.

### 4.2 Data Statistics

**Table 2: Top 10 web domain of Knowledge Pile**
| Web Domain | Count |
| --- | --- |
| en.wikipedia.org | 398833 |
| www.semanticscholar.org | 141268 |
| slideplayer.com | 108177 |
| www.ncbi.nlm.nih.gov | 97009 |
| link.springer.com | 85357 |
| www.ipl.org | 84084 |
| pubmed.ncbi.nlm.nih.gov | 68934 |
| www.reference.com | 61658 |
| www.bartleby.com | 60097 |
| quizlet.com | 56752 |

**Table 3: Main results of our model and baselines in some mathematical reasoning tasks and knowledge related reasoning tasks.**
|  |  | MATH | GSM8K | MMLU | AGIEval | BIG-Bench Hard |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 2 | 7B | 3.32 | 16.68 | 46.79 | 21.37 | 38.19 |
| Code-Llama | 7B | 3.88 | 14.4 | 40.39 | 21.47 | 42.78 |
| Minerva | 8B | 14.1 | 16.2 | - | - | - |
| Llemma | 7B | 14.3 | 35.94 | 47.89 | 24.14 | 48.61 |
| Llama 2-QoC | 7B | 6.2 | 28.51 | 57.02 | 30.04 | 44.82 |
| Qwen 2 | 7B | 10.82 | 51.4 | 57.97 | 40.37 | 22.27 |
| Baichuan2-Base | 7B | 5.72 | 23.81 | 53.95 | 34.68 | 40.32 |
| Mistral | 7B | 11.22 | 47.31 | 64.06 | 32.88 | 56.69 |
| Mistral-QoC | 7B | 17.48 | 55.27 | 65.71 | 45.24 | 57.81 |

Based on the proposed method, we have formed a high-quality knowledge dataset Knowledge Pile, which maintains about 735GB disk and 188B tokens (using Llama2 tokenizer). As shown in Figure 1, comparing with other datasets in academic and mathematical reasoning domains, we have acquired a large-scale, high-quality knowledge dataset at a lower cost, without the need for manual intervention. Through automated query bootstrapping, we efficiently capture the information about the seed query. Knowledge Pile not only covers mathematical reasoning data but also encompasses rich knowledge-oriented corpora spanning various fields such as biology, physics, etc., enhancing its comprehensive research and application potential.

Table 2 presents the top 10 web domains with the highest proportion of Knowledge Pile, primarily including academic websites, high-quality forums, and some knowledge domain sites. Table 4 provides a breakdown of the data sources’ timestamps in Knowledge Pile, with statistics conducted on an annual basis. It is evident that a significant portion of Knowledge Pile is sourced from recent years, with a decreasing proportion for earlier timestamps. This trend can be attributed to the exponential growth of internet data and the inherent timeliness introduced by the Knowledge Pile.

Figure: Figure 4: Timestamp Statistics of Knowledge Pile
Refer to caption: x5.png

## 5 Experiment and Discussion

### 5.1 Implementation Details

Based on Knowledge Pile, we further train two models: Llama2-QoC and Mistral-QoC, based on Llama2 and Mistral.

#### Training

In our experiment, we employed the InternLM(^1^11[https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM)) InternLM-Team () library for training all models on 256 A800 GPUs with bfloat16 mixed precision, and only utilized data parallelism during the training process. To enhance throughput and reduce memory consumption, we introduced the Flash attention 2 Dao () module. Above these, both Llama-QoC and Mistral-QoC achieve 4000 tokens per GPU per second(TGS). More training details will be described in Appendix A.

#### Evaluation

During the evaluation, we utilized the open-source library OpenCompass (^2^22[https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass)), which serves as a platform for evaluating large models. Leveraging Opencompass, we compare the performance with some open source pre-trained models: Llama2 Touvron et al. (), Code-Llama Rozière et al. (), Baichuan 2-Base Yang et al. (), Mistral Jiang et al. (), Qwen 2 Bai et al. () and some language models for mathematical reasoning: Llemma Azerbayev et al. () and Minerva Lewkowycz et al. (). For the selection of evaluation datasets, we opted for three distinct capabilities to assess both Llama2-QoC and Mistral-QoC. These encompassed mathematical reasoning datasets such as Math, GSM8K, knowledge-oriented language understanding datasets including MMLU, AGIEval, and challenging reasoning tasks BIG-Bench hard. More details for evaluation will be described in App B.

### 5.2 Main Results

Figure: Figure 5: The results in MMLU benchmark
Refer to caption: x6.png

The main results of our two models trained on Knowledge Pile ( Llama2-QoC and Mistral-QoC) and the baseline are compared in Table 3 across several general benchmarks. Overall, both models exhibit significantly improved performance, particularly Mistral-QoC. In the complex mathematical reasoning benchmark MATH dataset, Mistral-QoC demonstrates a notable enhancement after QoC training, rising from 11.22 to 17.48, which surpasses professional models such as LLEMMA and Minerva by 3 points (14.3 vs 17.48). Furthermore, Mistral-QoC achieves even higher performance on the mathematical application problem GSM8K (47.31 vs 55.27). Turning to various knowledge-based reasoning tasks, Mistral-QoC displays outstanding capabilities in both MMLU and AGIEval. On the challenging BIG-Bench Hard evaluation set, the model also exhibits a noteworthy improvement in handling complex reasoning tasks.

In comparison to the backbone model, LLAMA-QoC and Mistral-QoC show substantial improvements in mathematical and knowledge-based reasoning tests. For instance, Mistral achieves a 6-points improvement in the MATH dataset and a 5-point improvement in GSM8K. In knowledge-based tests, MMLU shows a relatively modest improvement, within a 1.7 point. However, in AGIEval, Mistral-QoC outperforms Mistral by an impressive 13 points.

Otherwise, an interesting observation is that when the baseline model performance is lower (e.g., LLAMA2), the enrichment of the dataset leads to higher improvements. Conversely, for baseline models with better performance, achieving significant improvement becomes relatively challenging. This difference may be attributed to variations in the model’s ability to fit the data.

### 5.3 Knowledge related reasoning

In order to compare the effects of Knowledge Pile on knowledge-related reasoning abilities, Figure 5 contrasts the improvement based on Llama2 and Mistral on MMLU, with a specific focus on performance across different subjects.

Figure: Figure 6: Some results in AGIEval benchmark
Refer to caption: x7.png

Figure: (a) Improving Llama2
Refer to caption: x8.png

From the perspective of Llama-QoC and Llama, following training with Knowledge Pile, the average performance on MMLU for all categories increased by at least 10 points. This notable performance enhancement can be attributed in part to the initially lower performance of Llama, while the Knowledge Pile exhibits relatively higher quality and rich knowledge content, playing a significant role in driving the improvement of Llama2’s performance.

In comparison to Llama, Mistral’s performance improvement is relatively moderate. The results indicate that in the STEM field, Mistral-QoC demonstrates a higher performance improvement compared to other fields, rising from 53.23 to 56.6. In contrast, performance improvements in other categories hover around one point. Two possible reasons account for this phenomenon. Firstly, Mistral exhibits strong comprehension abilities in disciplines such as humanities and social sciences, consistently scoring above 65, leaving relatively limited room for improvement. In contrast, the model’s score in the STEM field is only 53.23, suggesting a greater potential for performance enhancement. Secondly, Knowledge Pile has the largest and most extensive dataset in the STEM field, contributing to the more pronounced performance improvement of Mistral-QoC on MMLU-STEM.

Figure 6 compares the Performance of professional and academic exams covered in AGIEval. Even in the case of professional academic exams, Knowledge Pile similarly leads to substantial performance improvements. As observed in the figure, apart from a slight decline in the SAT-English expression test (from Mistral 68.93 to 65.53), other tasks exhibit noticeable improvements after continued training in Knowledge Pile. Notably, SAT-Math rises from 31.36 to 45, which is a significant improvement for Mistral. Legal exams such as JEC-QA-CA and JEC-QA-KD also show significant enhancements, with the performance increasing from 18.2 and 15.3 to 26.1 and 15.3. For other law-related exam, LSAT is divided into three aspects: Law-Analytics (LSAT-AR), Law-Logic(LSAT-LR), and Law-Reading(LSAT-RC) improvements of 2.17, 8.82, and 7.07 point, respectively. For the LogiQA, Mistral-QoC achieves an accuracy of approximately 45.01%, nearly a 20% improvement compared to Mistral (39.32).

### 5.4 Difference between Improve Llama and Mistral

As shown in Table 3 and Figure 5, it is evident that Llama2 exhibits inferior performance relative to Mistral. However, this also highlights a greater potential for performance improvement when training in Knowledge Pile. Also, we observe a significant behavioral difference during the improvement process, which is shown in Figure 7. We found that in certain tasks, Mistral’s performance undergoes a certain degree of decline during the improvement process, followed by a subsequent ascent after some time, eventually surpassing the previous performance levels. This phenomenon may come from a conflict between the potential distribution of the model and the distribution of high-quality datasets. Mistral’s potential distribution is superior but more unstable, whereas Llama’s performance is relatively poorer but exhibits greater plasticity. On the other hand, the improvement achieved on more powerful models (Mistral 7B) also demonstrates the high-quality of Knowledge Pile.

## 6 Conclusion

In this study, we propose an efficient method Query of CC, for the automated collection of specialized domain data. Leveraging seed data from some specific domains, we employ a large language model for query bootstrapping. By optimizing the breadth and depth of queries, we expand the query to retrieve data relevant to the specified domain. Ultimately, we collected and released an open dataset comprising approximately 735GB of Knowledge Pile, equivalent to approximately 188 billion tokens, in the fields of mathematics and knowledge. Experimental results demonstrate that the adoption of Knowledge Pile significantly enhances the model’s performance in some reasoning tasks, such as math word problems and professional examinations. Our objective is not only to establish a research foundation for community studies in mathematical and knowledge-related reasoning but also to provide an efficient and cost-effective method for collecting high-quality data, thereby facilitating the accumulation of more high-quality data.

## 7 Limitation

In this paper, we propose a collection method for domain-specific data, Query of CC, which extends domain-relevant queries through LLM bootstrapping for data retrieval. Additionally, we demonstrate through empirical evaluation on further training of LLAMA and Mistral that data collected using this method significantly improves the ability of LLM in some specific domains. However, this method still has the following potential limitations:

#### Data Quality

The quality of data collected by Query of CC depends largely on the quality of data in public corpora. In this work, we utilized the Common Crawl corpus as our public corpus, extracting and processing the CC dump up to April 2023. Despite leveraging some methods Wenzek et al. () for processing high-quality web data, we found that the corpus still contains an amount of low-quality and erroneously extracted content. Thus, improving the data quality of public corpora is also a direction for future research.

#### Data Scale

The data source of Query of CC is limited by public corpora, and unable to collect information not present in these corpora. This limitation requires particular attention. Moreover, the proportion of domain data in public databases also influences the scale of the collected dataset. During the collection process, scoring based on similarity scores can be employed to filter highly relevant data.

## Appendix A Model Training Details

In this study, we train all models with bf16 mixed precision and only data parallel. We employed the AMSP Chen et al. (), shard optimizer state across 8 cards to reduce communication overhead. Simultaneously, data parallelism was employed for training. To enhance throughput and reduce memory consumption, we introduced the Flash attention 2 Dao () module. All models underwent training for 50,000 steps, with a global batch size of 4 million tokens per step, totaling 200 billion training tokens.

During the initial 2,000 steps of training, the learning rate was warmed up to the maximum, and then, at the end of training, it was decayed according to cosine decay to the specified minimum learning rate. Specifically, for Llama2-QoC, the maximum learning rate during training was set to 2e-5, and the minimum learning rate was set to 2e-6. For Mistral-QoC, the maximum learning rate during training was 5e-6, descending to 2e-7 at the end of training. In our experiment, Llama2-QoC and Mistral-QoC achieved a training throughput of approximately 4000 tokens per GPU per second(TGS). Despite the relatively higher training efficiency of Llama2-QoC, it still utilized 14,000 GPU hours.

## Appendix B Evaluation

During the evaluation, we utilized the open-source library OpenCompass (^3^33[https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass)), which serves as a platform for evaluating large models. OpenCompass offers various evaluation datasets and supports efficient task partitioning to maximize the utilization of computational resources. For the selection of evaluation datasets, we opted for three distinct capabilities to assess both Llama2-QoC and Mistral-QoC. These encompassed mathematical reasoning datasets such as Math, GSM8K, knowledge-oriented language understanding datasets including MMLU, AGIEval, and challenging reasoning tasks BIG-Bench hard. The details of evaluation datasets are as follows:

#### Math Hendrycks et al. ( 2021b )

Math datasets comprising 12,500 competitive mathematical problems spanning challenging areas such as algebra and number theory. During evaluation, we selected four questions as examples, and each illustrating complete steps of problem-solving approaches. We assessed the model-generated answers for equivalence with these golden answers.

#### GSM8k Cobbe et al. ( 2021 )

GSM8k datasets contain 8.5k high-quality grade school math word problems, the task requires the large language model answered to combine world knowledge and mathematical reasoning. In this task, following xxx, we provide four questions and the solution with more detail as examples and also evaluate the equivalence of generated answer with the golden answer.

#### MMLU Hendrycks et al. ( 2021a )

MMLU is a vast multi-task dataset encompassing questions from various disciplines, including humanities, social sciences, STEM, and others. There are 57 sub-tasks in MMLU including elementary mathematics, American history, computer science, law, etc. During the evaluation, we provided five example questions and their answers with relatively complete chains of thought, using this to judge whether the model could correctly select options. This task necessitates a broad range of world knowledge and problem-solving capabilities for large language models.

#### AGIEval Zhong et al. ( 2023 )

AGIEval is also a benchmark that has various categories, and is designed for accessing the performance of large language models in context with human-centric standardized exams. Compared to MMLU, this dataset has more comprehensive data sources and provides a better evaluation of cross-linguistic knowledge performance.

#### BIG-Bench hard Suzgun et al. ( 2023 )

BIG-Bench hard is a challenging subset of BIG-Bench, which is designed to evaluate the reasoning ability of large language models. This dataset comprises 23 BIG-Bench tasks covering different programming languages and domains. During the evaluation, we provide three examples of large language models and assess whether the model could accurately answer the presented problems.

## Appendix C Collection implement details

In the implement section, we primarily discuss three components, the public corpora (C.1), retrieval engine (C.2), and the post-processing settings (C.3).

### C.1 Public Corpora

With the development of large language models, public corpora have become increasingly rich, including the Pile, RedPajama, and Common Crawl. To ensure the diversity of data sources for retrieval, we constructed our retrieval corpus based on Common Crawl. Common Crawl is an open-source web crawler project containing all publicly available web pages from 2013 to the present. In theory, it encompasses a significant portion of the information present on the web. Due to hardware cost constraints, we utilized WARC format data from the years 2016 to 2023 to build our retrieval database. We performed extraction, filtering, and cleaning procedures on the data obtained from Common Crawl to ensure the quality of the retrieval dataset. The final retrieval database comprises several billion records, occupying a total of 50TB of disk space.

### C.2 Retrieval Engine

Retrieving data from billions of documents is highly challenging, and we need to calculate the relevant score between queries and each document in the retrieved database. To improve storage and retrieval efficiency, we built the retrieval engine based on Elasticsearch (ES). Elasticsearch is an open-source distributed search engine that employs distributed storage and inverted indexes, achieving data retrieval highly efficient. We selected the BM25 algorithmic as the relevance scorer because of its efficiency. For each query, we recall the top 1000 most relevant documents. Leveraging Elasticsearch’s efficient storage and indexing algorithms, we can complete a query retrieval within 100ms.

### C.3 Post Processing

To enhance the quality of Knowledge Pile, inspired by Wenzek et al. (), we conducted data quality filtering and deduplication in the post-processing stage.

In the data quality filtering phase, we manually selected some high-quality data as positive examples and randomly selected low-quality data from Common Crawl, such as poorly structured data and advertising data, as negative examples to train our scoring model. High-quality data mainly includes papers, books, and high-quality forum data. We chose BERT-base-uncased as the backbone to train the model and tested it on a subset of data to ensure its high usability. Finally, we scored the retrieved data and filtered out data with quality scores below 0.8.

In the deduplication phase, we employed the Minhash-LSH method. For hyperparameter selection, we computed the similarity scores based on 13 grams and set the similarity threshold to 0.8. Additionally, we set $num_perm$ to 128 to balance computation efficiency and deduplication performance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Beyond the tasks you evaluated were there any performance changes after you further trained the LLM with Retrieve-Pile?
soundness: 2
strengths: Generating high quality datasets for augmenting the capabilities of LLMs into less covered domains is high relevant for the open-source as well as professional community. Data is the key for LLM success and this paper aims to present contribute for this matter.
summary: This paper investigates the problem of bootstrapping domain knowledge from general public corpora in order to reduce cost and time for manual data collection of domain-specific corpora. Using manually defined seed the presented approach, Retrieve-from-CC, first identifies seed examples in a large input corpus using BM25. For every retrieved record a LLM generates questions and responses are augmented by Chain of Thought sequences. After a quality filtering step the approach outputs a domain-specific dataset.
weaknesses: *Approach*:
- seeds: A downside of the approach is that it needs good seeds as input. At the same time, general-domain knowledge bases (dbpedia, yago) cover almost all domains to some degree. While the data goes beyond simple key phrases, the domains are probably only covered partially. The authors should consider leveraging this partial knowledge for generating input data for the bootstrapping phase. This drops the manual input requirement and might improve the dataset further.

*Evaluation*:
- My biggest criticism of the paper is that the author didn't compare against domain-specific LLMs. The question: "How does a LLM trained over a corpus generated with Retrieve-from-CC compare against a domain-specific LLMs?" is highly relevant for this paper and is not answered. For instance, one could compare against LLMs from the [Open Medical-LLM Leaderboard](https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard). There are probably other such domain-specific resources. 


* Misc: Typo generted in Figure 2

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Which LLM is used for query generation module? Did you see any difference in the queries generated with various LLMs ?
Assumption is that LLMs are not great at domain specific tasks, how does that impact your automatic query generation. Did you analyze the quality of queries generated for the domains? 
Is it the same model that is used for query generation or a bigger LLM is used ?
soundness: 3
strengths: Peper described a method for collecting domain specific training data without manual intervention, which is beneficial for makings LLMs perform better for domains of interest and making them more suitable for practical applications for a domain of interest. Method being automatic and showing improvements over the base LLMs is promising and can be beneficial in gathering large data that LLMs need for their training. Empirical results show that data generated results in significant improvements using it in LLM training. Quality metric also show that curated data is of good quality. Data pile created shows improvements during per-training as well as further training existing open models, which shows that generated data is of good quality.
summary: This paper provides a method called Retrieve-from-CC to curate domain specific data to train large language models. It uses a two phased approach where initial query keywords are given by humans and a LLM is used to generate queries which are fed to retriever (BM25) to gather data that is relevant for a specific domain. Authors publish a benchmark, Retrieve-Pile, that is covering four domains that includes sciences, humanities, Social Sciences and Miscellaneous.  Paper shows that using this data set helps in improving the performance on some of the mathematical benchmarks along with standard language benchmarks.
weaknesses: I see some places Knowledge-Pile is used without it being talked about anywhere. I guess its a typo instead of Retrieve-Pile it is used in evaluation section, tables, figures etc. This needs to be corrected. Lot of places I see space missing after Retrieve-Pile and other typos needs to be corrected as well.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: My suggestion for improvement would be to include more details about the Query Bootstrapping method and the metrics reported in the empirical section. I am confident that the language issues could be easily resolved as well (maybe with the help of an LLM even).
My main issue, however, is the unclear focus of the paper. For it to be a good resource/method paper, the data collection process should be described in more detail. For it to be a good empirical study paper, the experiments should reflect the argument of automating the dataset collection process vs. manual creation of a dataset. Unfortunately, this would require major revisions and potentially additional work. (The result however, might be two good papers (one with focus on the data, and another one with focus on the empirical evaluation), as the underlying questions are relevant and interesting)
soundness: 3
strengths: + extensive experiments, showing that their proposed data is helpful for training “smaller” (i.e. 7B Parameter) LLMs on domain specific task
+ automating the process of curating high-quality domain specific data
+ addressing the issue of data contamination 

The authors propose a method of automating the process of curating high-quality data in specific domains. Furthermore, they showcase how their data can be used to fine-tune LLMs and improve their performance on benchmark tasks within the respective domain.
summary: The authors propose Retrieve-from-CC, a data collection pipeline consisting of a a) query generation process and b) a document retrieval process based on the generated queries. They argue that their proposed method makes it possible to automate the data collection process for high quality domain-specific data. 
The so created dataset is then evaluated with regard to its composition (sources and domains) and its data quality (quantified as QuRating).
Finally, the authors use their newly composed dataset to fine-tune two LLMs (based on Llama2 and Mistral) and to train a Llama model from scratch. These models are then evaluated with regard to their performance on various benchmark datasets on mathematical and knowledge oriented language understanding tasks.
Their experiments indicate that their newly collected dataset of domain-specific high quality data can be used for fine-tuning LLMs and improve their performance on tasks that require specific knowledge. They additionally show that there is little to no dataset contamination when comparing the downstream task data with their own data.
weaknesses: 1) writing and language throughout the paper needs improvement. This makes the present work difficult to follow at times:

l. 67 “Otherwise, […]” -> additionally?
l. 70 “[…] continuing learning […]” -> unclear if they talk about fine-tuning
l. 82 “[…] for the quality and statistical of Retrieve-Pile […]” -> statistics?
l. 82 “We statistic […]” -> usage as a verb?

Paragraph starting at l. 179: Is question evolution the same as question extension from the previous paragraph?

l. 237 “Otherwise, we discuss about the different when improving different […]”

Unfortunately, these are only some highlighted examples where the quality of presentation is lacking.

2) exposure of results:

Table 4: It would be beneficial to mention the reported metrics. If not here, then at the description of the benchmark datasets.

This issue is persistent in most of the evaluation section. The authors report an increase of performance in “points” on multiple occasions, which frankly speaking could mean anything.

In the case of data quality, a little more in-depth explanation of the QuRating would be helpful in understanding the results. As this is by now not a wide-spread metric, it would be beneficial to explain how the values are obtained and what exactly they mean.

Overall, the authors could improve the exposition of results by providing a more detailed explanation of the used metrics, as this is an important bit of information for the reader.

3) focus:
Overall, the focus of the paper is not very well defined. First, the authors introduce a dataset collection method and provide a detailed overview of the collected dataset. For the present work to be a resource paper, the proposed Query Bootstrapping methods is not explained in sufficient detail. The paragraphs on “Question Extension”, “Thought Generation” and “Query post processing” are rather vague.

The other side of the spectrum would be a paper on an empirical study. For this to be the case, the evaluation section would require to be more detailed (regarding reported metrics).
In addition to that, following one of the author’s main argument (automating the process of collecting high-quality domain-specific data), it would be nice to see how LLMs that are trained on their data perform vs. LLMs that are trained on hand-crafted datasets.

My overall impression is that the authors should focus on either the resource side of their work, or the empirical side. In its current state, the lack of focus in combination with a (at times) poor representation makes the paper appear inconsistent and at times hard to follow.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: ### Suggestions

- Given the fully automated nature of the proposed pipeline, the resulting dataset is more like a synthetic dataset. This raises a different research question that how to generate high-quality synthetic datasets effectively and how this method compares to existing synthetic data generation approaches.

- I recommend that the authors invest effort in refining the paper's writing, as several language issues currently affect the fluency of reading.

- I recommend ensuring consistency in terminology throughout the paper. For instance, terms like 'Retrieve-Pile' and 'Knowledge Pile' appear to refer to the same dataset, which could cause confusion.
soundness: 2
strengths: - This work addresses a well-motivated goal: developing an accurate and scalable approach to extract training data from the evolving web, which is essential for keeping LLMs up-to-date.

- Experimental results show clear improvements in LLM performance on the listed benchmarks when further trained on the proposed dataset.

- The authors provide a detailed analysis of the proposed dataset, including multiple evaluation factors. Notably, the data leakage analysis between the pre-training dataset and the evaluation benchmarks is a valuable addition, helping to ensure the integrity of the results.
summary: This work presents an automated pipeline to collect domain-specific data from public corpora by leveraging large language models (LLMs) for query expansion and BM25 for efficient retrieval. The resulting dataset spans multiple domains, including STEM, humanities, social science, and Misc. While the method emphasises scalability and cost-effectiveness over manual curation, it also faces challenges in ensuring data quality and distinctiveness compared to existing human-curated datasets. Experimental results indicate that models trained on the proposed dataset are improved in several reasoning benchmarks.
weaknesses: ### About Novelty

- The authors have not clearly demonstrated how the proposed dataset differs from or adds unique value compared to existing high-quality, human-curated datasets, including the ones shown in Table 1.

- Additionally, the proposed pipeline lacks distinctive features that would make this automated construction process stand out as an innovative or superior alternative. For example, since this work focuses on domain-specific knowledge, it can be beneficial to leverage knowledge bases  or other kinds of structured data to help improve the relevance and accuracy of the data points.

### About Methodology
- While the authors emphasise scalability, fully relying on LLMs to refine and expand queries is both computationally expensive and prone to errors.

- Although BM25 offers efficiency and scalability in the retrieval phase, it does not guarantee high accuracy in the retrieved `(query, answer)` pairs. Even if standard dense retrieval techniques were employed, achieving consistently high accuracy would remain challenging.

- Consequently, errors introduced during both the query generation and retrieval phases could propagate, potentially compromising the overall quality of the final dataset.

### About Evaluation

- Given that LLM pre-training typically involves a broad set of evaluation benchmarks, this work lacks an analysis of potential "forgetting" on benchmarks (e.g., HELM, GLUE, LLM leaderboard) not included in the listed experiments.

- When comparing with other pre-training data, if you aim to demonstrate that this automatically generated dataset holds value even against human-curated datasets, it would be essential to directly compare their performance. Ideally, this comparison would show that the proposed dataset lags only by a small margin. Alternatively, comparing it with well-established synthetic data generation methods (also discussed in Suggestion 1) would also help substantiate the dataset's value and highlight areas for potential improvement.

Meta-review: **Summary**

Finding reliable data for LLMs is the challenge this paper focus on. The authors propose Retrieve-from-CC, a data collection pipeline consisting of a a) query generation process based on existing language models and b) a document retrieval process based on the generated queries. LLMs are used as query expansion.

**Strengths**

The goal of the paper is extremely important as find reliable data is one of the big challenges in improving LLMs

**Weaknesses**

- An evaluation of the method in term of quality of the data does not seem to be provided.
- The retrival cretiria should be expanded and explained.

**Final remarks**

The paper fails to convince reviewers that have interacted with the authors.

Decision: Reject