REInstruct: Building Instruction Data from Unlabeled Corpus (2024)

Abstract: Manually annotating instruction data for large language models is difficult, costly, and hard to scale. Meanwhile, current automatic annotation methods typically rely on distilling synthetic data from proprietary LLMs, which not only limits the upper bound of the quality of the instruction data but also raises potential copyright issues. In this paper, we propose REInstruct, a simple and scalable method to automatically build instruction data from an unlabeled corpus without heavy reliance on proprietary LLMs and human annotation. Specifically, REInstruct first selects a subset of unlabeled texts that potentially contain well-structured helpful and insightful content and then generates instructions for these texts. To generate accurate and relevant responses for effective and robust training, REInstruct further proposes a rewriting-based approach to improve the quality of the generated instruction data. By training Llama-7b on a combination of 3k seed data and 32k synthetic data from REInstruct, finetuned model achieves a 65.41% win rate on AlpacaEval leaderboard against text-davinci003, outperforming other open-source, nondistilled instruction data construction methods. The code is publicly available at https: //github.com/cs32963/REInstruct.

heading: 1 Introduction
text: Instruction Tuning (Ouyang et al., 2022) is an important step in aligning large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023a). By tuning LLMs on <instruction, response> pairs, instruction tuning enhances zero-shot performance of LLMs across a variety of tasks (Wei et al., 2022; Chung et al., 2022), and paves the way for further preference training such as reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Touvron et al., 2023b) and direct preference optimization (DPO) (Rafailov et al., 2023).
* Corresponding authors.
However, it is very difficult and costly to manually annotate high-quality instruction data (Köpf et al., 2023). Previous methods for automatic annotation largely rely on powerful proprietary LLMs like ChatGPT (OpenAI, 2023a), using techniques such as Self-Instruct (Wang et al., 2023) and EvolInstruct (Xu et al., 2023a). While these methods can generate large amount of instruction data and facilitate research in instruction tuning (Liu et al., 2023a; Chen et al., 2023), the upper bound of the quality of generated data is limited by the power of proprietary LLMs.
In this paper, we propose a simple and scalable method to automatically build instruction data from unlabeled text corpus. The proposed method requires only small amount of human-labeled data, and does not rely heavily on powerful proprietary LLMs. Our key motivation is that unlabeled text corpus contains a subset which is filled with highquality instruction following information, and converting these texts into proper instruction follow-
6840
ing format is much easier than labeling them from scratch. Specifically, we first employ a set of heuristic rules to obtain unlabeled text containing wellstructured helpful and insightful content. Given these content, we then generate related instruction to these high-quality content using an LLM reversely trained on a small set of high-quality human-annotated seed instruction data. Finally, given the generated instruction and the content, we employ a response rewriting process on the content, in order to obtain the high-quality and accurate response of the instruction from the content. To this end, we leverage a rewriting process in the form of generative machine reading comprehension, which is conducted by an addtional LLM trained on a small set of rewriting data.
To demonstrate the effectiveness of our automatic labeling method, we train LLMs jointly on our synthetic data and seed data across different data scales. We find that, Llama-7b (Touvron et al., 2023a) model finetuned with our largest 32k synthetic data and 3k seed data achieves 65.41% win rate on AlpacaEval leaderboard, outperforming other open-source, non-distilled methods with the same pre-trained checkpoint. This demonstrate the effectiveness of REInstruct.
Generally, the main contributions of this paper are:
• We propose a method to automatically build high-quality instruction data from unlabeled corpus, which requires only a small set of seed instruction data and a small set of rewriting data.
• We identify several heuristic rules to obtain high-quality unlabeled text for constructing instruction-following data, and propose a generative machine reading comprehension-based rewriting process to obtain high-quality response for each instruction.

heading: 2 Related Work
text: Instruction Tuning. Early works on instruction tuning focus on traditional NLP tasks, and instruction tuned LLMs show improved zero-shot performance across a variety of NLP tasks (Wei et al., 2022; Chung et al., 2022; Mishra et al., 2022; Sanh et al., 2022). Since the release of ChatGPT (OpenAI, 2023a), there have been increasing interests in general purpose instruction tuning, which expands instructions beyond NLP tasks (Ouyang
et al., 2022; Wang et al., 2023; Taori et al., 2023; Chiang et al., 2023). General purpose instruction tuning serves as the first step to build powerful LLM-based AI assistant (Ouyang et al., 2022; OpenAI, 2023a,b), providing foundation for further alignment training such as RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2023).
Automatic Instruction Data labeling. Manually annotating instruction data is very difficult and expensive, making it hard to scale (Köpf et al., 2023). Early instruction datasets are obtained by converting existing task-specific NLP datasets into instruction format (Wei et al., 2022; Chung et al., 2022; Sanh et al., 2022). Recent works have largely rely on powerful proprietary LLMs (Ouyang et al., 2022; OpenAI, 2023a) to generate high-quality instruction data. Alpaca (Taori et al., 2023) use SelfInstruct (Wang et al., 2023) to collect instruction data from InstructGPT (Ouyang et al., 2022) using its in-context learning ability (Brown et al., 2020). WizardLM (Xu et al., 2023a) devise Evol-Instruct to generate more complex and difficult instructions by prompting ChatGPT (OpenAI, 2023a). While these works (Ding et al., 2023; Chiang et al., 2023; Wu et al., 2024) can generate large amount of instruction data and help facilitate research in instruction tuning (Liu et al., 2023a; Chen et al., 2023), they rely on the underlying proprietary LLMs to ensure the diversity, difficulty and quality of instruction data, and fail to address the difficulty of annotating instruction data for these powerful proprietary LLMs.
There have been some effort in automatic instruction dataset annotation without reliance on powerful proprietary LLMs, trying to build instruction data using unlabeled text (Gu et al., 2022). Closest to our work is Humpback (Li et al., 2023a), which proposes to add instructions to unlabeled text and employ an iterative self-rating and selection pipeline to curate high-quality instruction data. However, Humpback relies on the existence of high quality unlabeled text already in the form of instruction response, and leaves the noise in unlabeled text and during instruction generation unaddressed, whereas we tackle the noise directly, resulting in a more efficient and robust pipeline.
Simulated Human Supervision. Since human supervision is difficult and expensive to obtain, recent works have explored the possibility of training LLMs to generate pseudo human supervisions, which usually helps reduce the annotation cost and speed up the alignment process. RLHF (Ouyang
et al., 2022; Touvron et al., 2023b) train LLMs on human preference data to provide reward signals for reinforcement learning to align LLMs. CoachLM (Liu et al., 2023b) use expert-annotated instruction revision data to train an LLM for automatic instruction data optimization. Our work is related as we train LLMs to generate instructions for unlabeled text and rewrite the response, which is essentially simulating how humans would annotate instruction data given candidate unlabeled text.

heading: 3 Method
text: The goal of our method is to automatically build instruction data from an unlabeled text corpus. In this section, we first outline how we select unlabeled texts that contain well-structured helpful and insightful contents (§3.1). Next, we describe how to convert these contents into <instruction, response> pairs through instruction generation (§3.2) and response rewriting (§3.3). Finally, we explain how to instruction finetune pretrained LLMs using both seed instruction data and our synthetic data (§3.4). Figure 2 illustrates the overall framework of our method.

heading: 3.1 Unlabeled Text Selection
text: The first step of our method is to select a subset of unlabeled text rich in high-quality instruction following information to steer the direction of model generation. Therefore, we aim to find a subset of text that contains well-structured helpful and insightful content to serve as the source of instruction tuning signals of our annotated data.
While there might be more sophisticated ways to select the desired text, we simply use a set of heuristic rules to filter out low quality text. Using
rule-based filtering has its merits compared with advanced neural methods, as it is fast, stable, and interpretable. The exact sampling rules can be found in Appendix A.1. Some of the major factors and rules we consider include:
1. Text length, which is generally related to the quality of the text. Serious and complicated topics often require longer text to elucidate, and longer text usually contains more details. Thus we filter out texts that are too short.
2. Text structure, which usually indicates the overall quality of the text and the expertise of the author. Well organized content shows the seriousness of writing and professionalism of the author. We keep text that contains multiple paragraphs starting with verbs in imperative form or present participles.
3. Text style, as formal writing usually results in high-quality content. We filter out text with too many first-person pronouns such as "I", "my", "we", and "us", as these pronouns are more common in casual writing.
4. Promotional text indicators, since text corpus includes lots of advertisement, we filter out text containing punctuation commonly found in advertisement. We also filter out text with too many all-capitalized words, as they are also common in promotional text.
The resulting subset is a diverse set of highquality unlabeled text, more suitable for converting into instruction response than randomly sampled text. We find this subset mainly consists of helpful guidance for a particular task, provide solutions for certain problem, or elucidate some complicated
concepts. Such texts are more likely to have a proper instruction, and help reduce the noise during instruction generation. The natural diversity inherited from unlabeled corpus also resembles our daily needs, which is more suitable for instruction tuning an LLM-based AI assistant. Table 1 shows an instance of our selected unlabeled text. More examples can be found in Appendix D.

heading: 3.2 Instruction Generation
text: To obtain instruction-response pairs, we reversely train an LLM on a small set of human-annotated seed instruction data Dseed = {(xs, ys)}, and use the trained model Mreverse to add instructions to the selected unlabeled text (Li et al., 2023a; Köksal et al., 2024). The generated instruction is answerable by the unlabeled text or by part of it. This is similar to the back-translation setting in machine translation (Sennrich et al., 2016). The objective
Answer the question based on the web text provided. Your answer must be helpful, detailed, and polite. Hide the fact that your answer is actually based on the web text, i.e. answer directly.
Web Text: [Selected unlabeled text]
Question: [Reversely generated instruction]

heading: 3.3 Response Rewriting
text: to rewrite selected unlabeled texts based on generated instructions. We ask ChatGPT to hide the fact that the response is generated based on some given unlabeled text (see prompt in Figure 3). We find ChatGPT returns valid responses grounded in the given text in most cases, demonstrating the quality of our selected text and the validity of generated instructions. However, there are two cases when ChatGPT fails to rewrite the response properly:
1. ChatGPT fails to hide the fact that the response is generated based on the given unlabeled text, and produces responses including text segments such as "Based on the given web text", which is a leakage from our rewrite prompt. These failure cases are filtered out when training the rewriting model.
2. ChatGPT decides that the unlabeled text does not contain enough information to generate a proper response and refuses to answer the instruction. These responses usually start with "I am sorry" or "I apologize". We keep these failure cases when training rewriting model, as they provide additional defense against noise in our unlabeled text selection and instruction generation. This is similar to the inclusion of unanswerable questions in machine reading comprehension (Rajpurkar et al., 2018). After response rewriting, these cases can be filtered out using simple rules (Appendix A.2).
Both the instruction generation model and the rewriting model are trained at a small scale, with data size similar to seed instruction data. They are kept fixed when scaling up the size of generated data without further annotation cost. We collect the generated instruction xg and the rewritten response yr to construct our final synthetic data Dsynthetic = {(xg, yr)}. Table 1 shows an instance of generated instruction and the rewritten response.

heading: 3.4 Joint Fine-tuning
text: For the final instruction tuning, we train LLMs jointly on the seed instruction data and our synthetic data. To balance two sources of training data, we up-sample seed data according to the size of synthetic data, as it proves useful in backtranslation (Edunov et al., 2018). We append different prompts to these two types of instruction data for differentiation, similar to the setting of tagged back-translation (Marie et al., 2020). Specifically,
we use the same prompt from Humpback (Li et al., 2023a), i.e. we append Prompt Prseed = "Answer in the style of AI Assistant." after the seed instructions and append Prompt Praug = "Answer with knowledge from web." after our generated instructions. During inference, we prepend Prseed before Praug, as is shown to be beneficial for the final performance (Li et al., 2023a). The objective of joint fine-tuning is:
Lfinetune =λ ∑
(xs,ys)∈Dseed − log p(ys|Prseed, xs)
+ ∑
(xg ,yr)∈Dsynthetic − log p(yr|Praug, xg)
(3) where λ represents upsample ratio of seed data.

heading: 4 Experiments

heading: 4.1 Setup
text: Dataset. We use C4 (Raffel et al., 2020) as our unlabeled text corpus, since it is a free open-source dataset commonly used in pre-training LLMs. C4 contains around 400M text segments cleaned from Common Crawl Project1. For the seed instruction data used in training instruction generation model and final instruction tuning, we use 3k high-quality (rank 0) instruction data obtained from the first round of conversations in OpenAssistant (Köpf et al., 2023). We collect 4k rewriting data from gpt-3.5-turbo for training the rewriting model.
Baselines. We mainly compare our method with other instruction tuned models that do not rely on powerful proprietary models:
• OASST3K (Köpf et al., 2023): Llama model tuned with our 3k seed instruction data.
• Guanaco (Dettmers et al., 2023): Llama model tuned with 9k instruction data from all turns in OpenAssistant dataset.
• Dromedaryours (Sun et al., 2023): Llama model trained with 40k randomly sampled instruction data from the original Dromedary dataset.
• Humpbackours (Li et al., 2023a): We reimplemented Humpback using C4 and Llama7b under our settings. More details and discussions can be found in Appendix B.
1https://commoncrawl.org
We also compare with models trained with instruction data distilled from proprietary LLMs (Chiang et al., 2023; Xu et al., 2023a,b) to show the competitiveness of our method.
Evaluation. We use AlpacaEval (Li et al., 2023b) to evaluate the instruction following capability of our instruction tuned LLMs. AlpacaEval consists of 805 test instructions that covers a variety of tasks. The evaluation metric is the win rate against reference response by text-davinci-003, judged by GPT-4 (OpenAI, 2023b).
Implementation Details. We use Llama-7b (Touvron et al., 2023a) as our pre-trained base LLM for data construction, which is fine-tuned for instruction generation and response rewriting. For training these two models, we use a batch size of 128, a learning rate of 2e-5 and a cosine learning rate schedule for 10 epochs, which gradually reduces the learning rate to zero. The up-sample (Edunov et al., 2018) ratio for seed data during joint-tuning is chosen according to the growing size of newly generated data, keeping the ratio of generated data to seed data roughly 2:1 (see Appendix A.3). All instruction tuning experiments are conducted with the same batch size of 128 and a constant learning rate of 2e-5. We use 4 or 8 A100-80G GPUs for most of our training and carry out full-parameter tuning. During instruction gen-
eration and response rewriting, we use greedy decoding with repetition penalty (Paulus et al., 2018; Klein et al., 2017) of 1.05.

heading: 4.2 Main Results
text: We select unlabeled text of size 8k, 16k, and 32k for our experiments, and train Llama-7b jointly on seed data and our synthetic data across different data scales. Table 2 shows the results of our method and other models on AlpacaEval. We can see that:
1) REInstruct can achieve better performance than other open-source, non-distilled methods. Our method, which learns from only a few seed data and automatically built instructions, surpasses most non-distilled methods and achieves competitive performance with some distilled methods, such as Baize-v2-7B and Vicuna-7B-v1.1.
2) By automatically building high-quality instruction data from an unlabeled corpus, REInstruct can lead to scalable improvement on instruction following. Experiments with 8k, 16k, and 32k data show continuous performance gains as data volume grows.
To better verify the effectiveness of our model, we further compare its performance with other distilled models on commonsense reasoning. The evaluation is conducted using lm-eval (Gao et al., 2023). As shown in Table 3, our method achieves
better or competitive performance on five commonsense reasoning datasets.

heading: 4.3 Quality Analysis of Auto Generated Data
text: This section analyzes the quality of automatically generated data, including generated instructions and rewritten responses.

heading: 4.3.1 Diversity of Generated Instruction
text: Since our seed instruction data consists of just over 3k data points, we select a subset of 4k unlabeled text from C4 for instruction generation, maintaining a similar data size. Figure 4 shows the sunburst visualization of verb-noun structure of seed instructions and our generated instructions, as used in Self-Instruct (Wang et al., 2023) to show the diversity of tasks. We found our generated instructions are quite diverse and similar to the seed instructions, covering a wide range of queries that may be sent to an AI assistant for help or guidance.

heading: 4.3.2 Similarity of Unlabeled Text and Rewritten Response
text: To show that our rewriting process generates responses based on the unlabeled text and preserves the instruction following signals in it, we measure text similarity between the unlabeled text and the rewritten response. Specifically, we calculate the ratio of words in rewritten response that can be directly copied from unlabeled text. We found that 77.63% of the words in rewritten response also appear in the unlabeled text.

heading: 4.4 Detailed Analysis
text: This section analyzes the effects of Unlabeled Text Selection and Response Rewriting. We design four ablated variants of REInstruct:
• “w/o Unlabeled Text Selection” directly samples unlabeled text from the corpus at random, rather than using several heuristic rules to obtain high-quality texts (§4.4.1);
• “w/o Response Rewriting” removes the response rewriting step and directly uses unlabeled text as the response (§4.4.2);
• “Self-Rewrite” uses the LLM only trained on seed data to rewrite the unlabeled text (§4.4.2);
• “Self-Training” uses the self-generated response instead of the rewritten response from unlabeled text (§4.4.3);
We conduct analysis at the scale of 8k synthetic data, and Table 4 shows the win rate on AlpacaEval of these different variants.

heading: 4.4.1 Effect of Unlabeled Text Selection
text: To understand the effect of our unlabeled text selection, we compare it with random sampling from the text corpus. Figure 5 shows the sunburst visualization of instructions generated on randomly sampled unlabeled text. As seen, most instructions are limited to a few text generation tasks, indicating low instruction diversity.
We then conduct an instruction tuning experiment and evaluate on AlpacaEval, using the same process as the original REInstruct but with the randomly sampled unlabeled text. Table 4 shows that using randomly sampled unlabeled text (w/o Unlabeled Text Selection) leads to no significant performance improvement over the baseline OASST-7B3K (61.70% to 61.42%), while our method REInstruct-7B8K improves performance from 61.70% to 63.78%.
Based on the above observation, it is clear that not all unlabeled texts are worth converting into instruction tuning data. And our unlabeled text selection can effectively obtain high-quality unlabeled text for constructing instruction-following data.

heading: 4.4.2 Effect of Response Rewriting
text: To show the necessity of the rewriting process, we conduct instruction tuning without rewriting unlabeled text, i.e. the selected unlabeled text is di-
rectly used as instruction response. As shown in Table 4, using unlabeled text as responses (w/o Response Rewriting) significantly degrades performance. This observation indicates that the content from unlabeled texts are unsuitable to be directly used as accurate instruction responses for model training.
An interesting question is whether we can skip with the external rewriting data and directly employ the LLM instruction-tuned on the seed data to rewrite the unlabeled text. We refer such approach as Self-Rewrite. Table 4 shows that self-rewrite leads to inferior performance. We suspect that LLMs fine-tuned on a small set of seed data with relatively short instructions might struggle with the rewriting task where the rewriting instructions are much longer. Adding longer instructions or other content-grounded generation data to the seed data might help reduce reliance on external rewriting data. We will explore this in future work.

heading: 4.4.3 Effect of Instruction Following Signals in Unlabeled Text
text: In this section, we investigate the role of instruction following signals extracted in our selected unlabeled text. Previous works (Zhou et al., 2023) have shown that LLMs obtain quite reasonable instruction following capability with limited instruction data. Therefore, it is possible that a self-training approach where model is trained on its own responses on a larger set of instructions may lead to better performance.
To understand the difference between selftraining and our method, we use instructions reversely generated on our unlabeled text to prompt LLM tuned on seed data. We then finetune a new LLM jointly on the seed data and the self-generated data. In other words, the sampled unlabeled text is only used to produce a diverse set of instructions, and does not provide instruction following signals in the response.
Table 4 shows that the result of self-training is inferior to our method. Our hypothesis is that good instruction tuning requires continuously providing high quality instruction following signals that can improve current output space of the trained LLM. The instruction data generated in a self-training fashion already holds large amount of probability space, and does not provide much training signals.

heading: 5 Conclusion
text: In this paper, we propose REInstruct, a simple and scalable method to automatically build instruction data from unlabeled corpus. Our method only requires small amount of labeled data, and does not heavily rely on proprietary LLMs. Experiment results show that instruction tuning LLMs on a small set of human-labeled data and our synthetic data achieves competitive instruction following performance, demonstrating the quality of our synthetic data. We believe this work provides insights for instruction data annotation and helps foster research in instruction tuning. For future work, we aim to develop more sophisticated neural methods to collect and extract instruction following signals from unlabeled dataset across different modalities.
Limitations
Using heuristic rules to sample unlabeled text has certain limitations. First, it only captures some shallow textual features that are indicators of highquality text, with limited semantic understanding of the content. Second, it might be hard to generalize to other data distributions such as code, or other modalities such as vision. Finding high-quality unlabeled data might be more subtle in these settings.
While our method leads to more capable LLMs, further alignment training is required to reduce the hallucination and toxicity of the instruction tuned LLMs.

heading: Acknowledgements
text: We sincerely thank all anonymous reviewers for their insightful comments and valuable suggestions. This research work is supported by the National Natural Science Foundation of China under Grants no. 62122077, no. 62306303, no. 62106251, and no. 62076233.

heading: B Our implementation of Humpback
text: We use the exact scoring prompt in Humpback (Li et al., 2023a), and run the iterative curation process on our 32k selected unlabeled text. The base model we use is Llama-7b. We found that during the first iteration, about half of our unlabeled text is ranked 5, and the other half is ranked 4. We use texts with score 5 to train the model in the second iteration, and the resulting model produces score 5 for almost all our unlabeled text, thus the performance of the final trained model will be the same as our ablation experiment "w/o Response Rewriting" in Table 4, which has win rate about 50%, much lower than the performance reported in the Humpback paper. Possible reasons for such performance are:
1. Difference in implementation: Since Humpback is not open-source, some exact details are unclear. For example, it is possible that they use Llama-65b model to curate instruction data and use the obtained data to train Llama-7b model for ablation. We also notice other self-alignment works (Sun et al., 2023; Yuan et al., 2024) typically use larger LLMs of scale 65b or 70b.
2. Difference in unlabeled dataset: Since ClueWeb (Overwijk et al., 2022a) used in Humpback (Li et al., 2023a) is not a free open-source dataset, and we are unable to obtain it in a short time, we use the more accessible C4 dataset for our experiment. The difference between CommonCrawl and ClueWeb is well-discussed in Overwijk et al. (2022b), and ClueWeb generally contains higher quality texts that are more likely to satisfy potential information needs.
To the best of our knowledge, there are no public implementations of Humpback on the 7b model scale. On the other hand, it is completely possible to combine self-alignment works such as Humpback (Li et al., 2023a) with our method. We leave that for future investigation.

heading: C Possible Bias from GPT-4 Evaluation
text: In the case of AlpacaEval, three potential biases of automatic evaluators such as GPT-4 are:

heading: D Examples
text: We found almost every unlabeled text we selected contains some useful information or knowledge. However, it is possible that average people like us are not capable to judge whether the text contain the best information or knowledge about the discussed topic. Table 7 provides an example that we think might not be good enough. Meanwhile, some texts that are suitable for converting into instruction data might be filtered out by our heuristic rules. We consider finding such texts a major future direction. One possible solution is to use the selected texts as positive examples and random texts as negative examples to train a neural classifier. Table 8 shows an example that might be good for converting into instruction data.
We include more <unlabeled text, generated instruction, rewritten response> triples in Table 9 and Table 10, so that readers can better understand our synthetic data.

heading: Future Work
text: We consider finding such texts a major future direction. One possible solution is to use the selected texts as positive examples and random texts as negative examples to train a neural classifier. Table 8 shows an example that might be good for converting into instruction data. We include more <unlabeled text, generated instruction, rewritten response> triples in Table 9 and Table 10, so that readers can better understand our synthetic data