: An Easy-to-use Instruction Processing Framework for Large Language Models (2024)

Abstract: In recent years, instruction tuning has gained increasing attention and emerged as a crucial technique to enhance the capabilities of Large Language Models (LLMs). To construct highquality instruction datasets, many instruction processing approaches have been proposed, aiming to achieve a delicate balance between data quantity and data quality. Nevertheless, due to inconsistencies that persist among various instruction processing methods, there is no standard open-source instruction processing implementation framework available for the community, which hinders practitioners from further developing and advancing. To facilitate instruction processing research and development, we present 1, an easy-to-use instruction processing framework for LLMs, which modularizes instruction generation, selection, and prompting, while also considering their combination and interaction. EasyInstruct is publicly released and actively maintained at https://github.com/zjunlp/ EasyInstruct, along with an online demo app2 and a demo video3 for quick-start, calling for broader research centered on instruction data and synthetic data.

heading: 1 Introduction
text: Large Language Models (LLMs) have brought about a revolutionary transformation in the field of Natural Language Processing (NLP), leading to substantial improvement in performance across various tasks (Brown et al., 2020; OpenAI, 2023; Anil et al., 2023; Touvron et al., 2023b; Zhao et al., 2023; Chen et al., 2022; Qiao et al., 2023; Chen,
∗ Corresponding Author. 1This is a subprobject of KnowLM (https://github.
com/zjunlp/KnowLM), which facilitates knowledgeable LLM Framework with EasyInstruct, EasyEdit (Wang et al., 2023a; Yao et al., 2023; Zhang et al., 2024), EasyDetect etc.
2https://huggingface.co/spaces/zjunlp/ EasyInstruct
3https://youtu.be/rfQOWYfziFo
2023). To optimize the performance of LLMs in specific tasks or domains, it is crucial to adapt their outputs to specific contexts or instructions. Recent studies (Wei et al., 2022; Ouyang et al., 2022; Chung et al., 2022) have proposed instruction tuning methods for fine-tuning LLMs, which is a prominent research area aimed at optimizing the LLMs’ behavior by providing explicit instructions during training, enabling better control and alignment with user preferences and desired outputs. Instruction dataset construction, which is also referred to as data engineering or management, poses a significant challenge in the process of instruction tuning (Zhao et al., 2023; Zhang et al., 2023; Wang et al., 2023c,d).
Substantial efforts have been dedicated to the task of construction instruction data through human annotations (Wang et al., 2022; Köpf et al., 2023), requiring a significant allocation of resources. Against this backdrop, LLMs are utilized to synthesize large-scale instruction data automatically (Wang et al., 2023b; Xu et al., 2023; Li et al., 2023b). These methods could scale up the size of instruction-following data, but they still inevitably suffer limited diversity and complexity, resulting in an unbalanced distribution and poor quality of instruction data. Recent studies (Zhou et al., 2023; Chen et al., 2023a; Xu et al., 2023) have unveiled a seminal revelation, indicating that even a small quantity of high-quality instruction data has the potential to yield robust performance. In general, instruction processing is an important process requiring careful attention to detail and rigorous quality assurance procedures to construct a high-quality instruction dataset for LLMs.
Unfortunately, the availability of open-source tools for instruction processing remains limited, especially in comparison to many opensource projects on models and training infrastructures (Touvron et al., 2023a,b; Taori et al., 2023; Scao et al., 2022; Chiang et al., 2023; Zeng
94

heading: Corpus
text: et al., 2023). Existing projects are often highlycustomized to their own needs, lacking a systematized and modular processing ability to address diverse processing pipelines for LLMs. For instance, the Alpaca (Taori et al., 2023) dataset targets the augmentation of diversity for LLaMA tuning, whereas AlpaGasus (Chen et al., 2023a) focuses on filtering out low-quality instances from Alpaca. Thorough development of instruction processing systems for the ever-evolving and emerging requirements of LLM remains unexplored, particularly in light of the quick expansion of inventive LLM applications spanning various fields.
To address this issue, we develop EasyInstruct as depicted in Figure 1, an easy-to-use instruction processing framework for LLMs. Given some existing chat data, corpus, or Knowledge Graphs, EasyInstruct can handle instruction generation, selection, and prompting processes, while also considering their combination and interaction. These consistencies facilitate further development and comparisons of various methods, thus promoting the advancement of better instruction processing work. We further conduct experiments with EasyInstruct to validate its effectiveness in instruction processing. Currently, EasyInstuct is open-sourced on GitHub and has already received over 300 stars. We are committed to the long-term maintenance
of EasyInstruct, providing continuous support for new features to ensure its effectiveness as a framework for instruction processing and synthetic data generation (Bauer et al., 2024).

heading: 2 Background
text: LLMs typically undergo two stages of training: pretraining and fine-tuning (Zhao et al., 2023). Despite the fact that large-scale pretraining is the key of the model’s proficiency in generating natural language responses, these pre-trained models can still struggle with comprehending human instructions accurately. To bridge the gap between the training objectives and human objectives, instruction tuning is introduced as a potent strategy to amplify the controllability and capabilities and of LLMs in interpreting and responding to instructions (Wei et al., 2022; Ouyang et al., 2022; Chung et al., 2022; Wang et al., 2023b; Zhang et al., 2023; Lou et al., 2023). Concretely, instruction tuning involves the method of refining pre-trained LLMs through supervised learning, utilizing examples structured as (INSTRUCTION, INPUT, OUTPUT). In this format, INSTRUCTION represents the human-given directive that outlines the task, INPUT optionally offers additional context, and OUTPUT signifies the expected outcome in alignment with the INSTRUCTION and any given INPUT.
Despite the effectiveness of instruction tuning, constructing high-quality large-scale instructions which effectively encompass the target behaviors remains a non-trivial challenge in this realm. Existing instruction datasets are often limited in terms of diversity, quantity, and creativity, which underscores the significance of instruction processing. One typical method for constructing instruction datasets is data integration. In this method, instructional datasets are constructed by merging existing annotated datasets with descriptions of tasks in natural language (Longpre et al., 2023; Sanh et al., 2022; Anand et al., 2023). Another prevalent method for constructing instruction datasets is automated generation. To alleviate the need for extensive human annotation or manual data gathering, automated methods have been proposed to generate large volumes of instructional data through the use of LLMs. Instructions can be sourced from chat data (Chiang et al., 2023) or expanded on a small set of seed instructions using LLMs (Wang et al., 2023b; Xu et al., 2023; Li et al., 2023b). Subsequently, the collected instructions are fed into LLMs to generate corresponding inputs and outputs. In EasyInstruct, our primary focus lies on automated approaches for instruction generation due to their high efficiency and scalability.
Another promising research direction of instruction processing is the selection of high-quality instruction. Recently, numerous studies (Zhou et al., 2023; Chen et al., 2023a; Xu et al., 2023; Liu et al., 2023) have investigated the issue of the scale of the instruction dataset for fine-tuning and have indicated that merely increasing the number of instructions may not necessarily result in enhancements. Instead, a modest volume of high-quality instruction data can influence the fine-tuning of LLMs, yielding solid performance. Thus, optimizing the instruction dataset and enhancing its quality play a critical role in fine-tuning LLMs effectively.
From a practical implementation point of view, instruction processing is actually complex and requires meticulous consideration. In this paper, we present , an easy-to-use framework to effectively and efficiently implement instruction processing approaches including instruction generation, selection, and prompting. Through this framework, EasyInstruct can help users to quickly comprehend and apply the existing instruction processing methods implemented in the package.

heading: 3 Design and Implementation
text: As illustrated in Figure 1, EasyInstruct provides a complete instruction processing procedure built on PyTorch and Huggingface. In this section, we first introduce the design principles, and then detail the implementation of the major modules.

heading: 3.1 Design Principles
text: The framework is designed to cater to users with varying levels of expertise, providing a userfriendly experience ranging from code-free execution to low-code customization and advanced components extension options:
Zero-Code Instruction Processing. Novice users, who do not require coding knowledge, can leverage pre-defined configuration files and shell scripts to accomplish code-free instruction processing. By running these scripts, they can complete instruction processing tasks without the need for coding skills. Example configuration files and shell scripts are shown in Appendix A.2.1.
Low-Code Customization. Intermediate users have the option to customize various process inputs and outputs using a low-code approach. This allows them to have more control over the different stages within the framework. A running example is shown in Figure 2.
Advanced Components Extension. Experienced users can easily extend our components based on their specific scenarios and requirements. To customize their classes, users can inherit the base classes of modules and override the necessary methods as per their requirements. This flexibility enables them to implement their functional components, tailored to their unique needs.

heading: 3.2 APIs & Engines
text: The APIs modules integrate with mainstream LLMs, including API services provided by companies such as OpenAI4, Anthropic5, and Cohere6. This integration facilitates the seamless invocation of various relevant steps within the framework. We list a range of API service providers and their corresponding LLM products that are currently available in EasyInstruct in Appendix A.5. The Engines module standardizes the instruction execution process, which enables the execution of
4https://platform.openai.com/docs 5https://docs.anthropic.com/claude/docs 6https://docs.cohere.com/docs
instruction prompts on several open-source LLMs such as LLaMA (Touvron et al., 2023a,b) and ChatGLM (Du et al., 2022; Zeng et al., 2023).

heading: 3.3 Generators
text: The Generators module streamlines the process of instruction generation, enabling automated generation of instruction data based on seed data, where seed data can be sourced from either chat data, corpus, or Knowledge Graphs. As listed in Table 1, the instruction generation methods implemented in Generators are categorized into three groups, based on their respective seed data sources.
Chat Data. Early work (Wang et al., 2023b) randomly samples a few instructions from a humanannotated seed tasks pool as demonstrations and then, prompts an LLM to generate more instructions and corresponding input-output pairs. Due to its adaptability, Self-Instruct remains the prevailing preference among automated instruction generation methods. Similarly, starting with an initial set of instructions, Evol-Instruct (Xu et al., 2023) incrementally upgrades them into more complex instructions by prompting an LLM with specific prompts. In contrast to the Self-Instruct generation approach, Evol-Instruct allows for the adjustment of the difficulty and intricacy of the instructions it produces.
Corpus. Given an unannotated corpus, Instruction Backtranslation (Li et al., 2023b) creates an instruction following training instance by predicting an instruction that would be correctly answered by a paragraph in the document or corpus. Considering the mixed quality of human-written web text and the presence of noise in generated content, only the highest quality instances are reserved.
Knowledge Graphs. Incorporating existing knowledge graphs, KG2Instruct (Gui et al., 2023) generates Information Extraction (IE) instruction datasets. To enhance the generalizability of instructions, a random sampling approach is utilized based on human-crafted instruction templates.
EasyInstruct has implemented the existing methods above to facilitate future research and systematic comparison of automated generation of instruction data. Furthermore, the flexibility of the Generators module allows practitioners to select the appropriate generator and make further modification that best suits their specific needs. A running example of using a Generator class in EasyInstruct is shown in Figure 2.
and selection in .

heading: 3.4 Selectors
text: The Selectors module is designed to streamline the process of filtering instructions, enabling the curation of instruction datasets from raw instruction data. This raw data might originate from publicly accessible instruction datasets or be synthesised in advence by the Generators module. Table 1 provides a comprehensive overview of various metrics for instruction quality evaluation. We divide the evaluation metrics into four categories based on the principle of their implementation: statistics-based, n-gram-based, structure-based and LM-based. All Selector classes derive from a common base class, BaseSelector. It includes fundamental attributes and abstract methods such as loading, processing, and dumping of data. In EasyInstruct, multiple Selectors can be grouped for convenient usage, which allows users to achieve more concise and readable code. A running example of using a Selector class is shown in Figure 2.

heading: 3.5 Prompts
text: The Prompts module standardizes the instruction prompting step, in which user requests are constructed as instruction prompts and sent to specific LLMs to obtain responses. Utilizing the Prompts module with a series of well-designed and refined prompts enhances the ability of Generators and Selectors to effectively fulfill their respective functions. Similar to Selectors, all

heading: Modules Methods Seed Description

heading: Modules Metrics Type Description
text: Prompts classes inherit from a common base class, BasePrompt, which includes necessary attributes and abstract methods. In the mentioned base class, there are functionalities provided for building prompts, requesting generation results from LLMs, and parsing the responses received from LLMs. The base class also provides mechanisms to handle error conditions and exceptions that may occur during the whole process. Users can inherit from the base class and customize or extend its functionality based on their specific requirements. We also equip EasyInstruct with various prompting techniques and application adaptions (e.g. Chainof-Thought, Information Extraction, Multimodal, etc.) by providing a consistent and standardized interface, enabling efficient instruction prompting for LLMs.

heading: 4 Evaluation
text: In terms of evaluation, we will introduce the experiment setups and illustrate the empirical results of multiple modules implemented in EasyInstruct to demonstrate its capability.

heading: 4.1 Experiment Setups
text: Instruction Datasets. We adopt the popular SelfInstruct (Wang et al., 2023b) and Evol-Instruct (Xu
et al., 2023) methods implemented in EasyInstruct to synthesize instruction datasets, containing instructions paired with instance inputs and outputs separately. We mainly consider four instruction datasets as follows: (a) self_instruct_5k is constructed by employing the Self-Instruct method to distill instruction data from text-davinci003; (b) alpaca_data_5k is randomly sampled from the Alpaca dataset (Taori et al., 2023); (c) evol_instruct_5k is constructed by employing the Evol-Instruct method; (d) easyinstruct_5k is collected by integrating the three instruction datasets above and applying multiple Selectors in EasyInstruct to extract high-quality instruction datasets. We detail the hyper-parameters for selection in a configuration file, as shown in Appendix A.2.1.
Implementation Setups. To conduct the experiments on the effect of instruction datasets, we adopt a LLaMA2 (7B) model (Touvron et al., 2023b), a popular open-sourced LLM backbone widely used for instruction-tuning. We fine-tune the LLMs with LoRA (Hu et al., 2022) in the format following Alpaca (Taori et al., 2023) with KnowLM7. All hyperparameters for training can be found in Appendix A.3. The evaluation is conducted by comparing the
7https://github.com/zjunlp/KnowLM
generated results from different fine-tuned models based on the AlpacaFarm (Dubois et al., 2023) evaluation set. Following AlpacaFarm, for each comparison, we employ ChatGPT as the evaluator to automatically compare two outputs from different models and label which one they prefer, reporting the win rate as the evaluation metric. For both instruction tuning and evaluation, we adopt the same prompt templates used by Alpaca-LoRA8, as shown in Appendix A.4.

heading: 4.2 Experiment Results
text: Main Results. We compare the generated outputs from models fine-tuned separately on the four instruction datasets with the outputs from the base version of the LLaMA2 (7B) model on the AlpacaFarm evaluation set. As depicted in Figure 3, there are improvements in the win rate metric for all the settings. Moreover, the model performs optimally under the easyinstruct_5k setting, indicating the importance of a rich instruction selection strategy.
Instruction Diversity. To study the diversity of the instruction datasets considered in our experiments, we identify the verb-noun structure in the generated instructions and plot the top 20 most prevalent root verbs and their top 4 direct nouns
8https://github.com/tloen/alpaca-lora
in Figure 4, following the approach of Wang et al. (2023b). Overall, we see a wide range of intents and textual formats within these instructions.
Case Study. To conduct a qualitative evaluation of EasyInstruct, we sample several instruction examples selected by the Selectors module in easyinstruct_5k for the case study. We also attach the corresponding evaluation scores for each of these instruction examples, as shown in Table 2. We observe that the selected instructions often possess fluent language and meticulous logic.

heading: 5 Conclusion and Future Work
text: We present , an easy-to-use instruction processing framework for LLMs. EasyInstruct can combine chat data, corpus, KGs and LLMs as an automated instruction generation tool, reducing the cost of manual data annotation. Additionally, EasyInstruct integrates a diverse set of instruction selection tools to optimize the diversity and distribution of instruction data, thereby improving the quality of fine-tuning data. EasyInstruct is designed to be easy to extend, and we will continue to update new features (e.g., knowledgeable synthetic data generation) to keep pace with the latest research. We expect EasyInstruct to be a helpful framework for researchers and practitioners to facilitate their work of instruction tuning on LLMs.

heading: Limitations
text: In this paper, we are committed to unifying all phases of instruction data processing including instruction generation, selection, and prompting. Despite our efforts, this paper may still have some remaining limitations.

heading: The Scope of Instruction Selection Methods.
text: We implement various instruction selection methods within the Selectors module. Based on the evaluation metrics utilized and the model base employed, the implemented instruction data selection methods can be divided into three categories: meth-
ods based on a system of indicators, methods utilizing powerful LLMs like ChatGPT, and methods employing small models (Wang et al., 2024). However, another line of work (Li et al., 2023a,c,b; Wu et al., 2023; Chen et al., 2023b; Kung et al., 2023) employs trainable LLMs like LLaMA for computation formulas in instruction selection processes, which are not integrated into the Selectors module. Although our design choice is to decouple instruction processing and model training into two separate phases, we regard it as a limitation that may be addressed by future work.
Statistics for evaluating efficiency. In our evaluation, we fine-tune a LLaMA2 (7B) model utilizing multiple modules implemented in EasyInstruct. Compared to models fine-tuned on other instruction datasets constructed without EasyInstruct, our model achieves optimal results, demonstrating EasyInstruct’s capability. Although we also qualitatively demonstrate the ease of writing code for instruction processing with multiple code samples and configuration files using EasyInstruct, a limitation is the lack of appropriate statistics for quantitatively evaluating efficiency.

heading: Acknowledgments
text: We would like to express gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), CCF-Baidu Open Fund, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

heading: A Appendix
text: A.1 Installation
Currently, EasyInstruct offers three installation options, each accompanied by its corresponding installation script. Users can choose the option that best suits their specific requirements.
A.1.1 Installation from GitHub Repository The first option is to install the latest version of EasyInstruct from the GitHub repository. The installation script is shown in Figure 5.
A.1.2 Installation for Local Development The second option is to download the source code for local development. The installation script is shown in Figure 6.
A.1.3 Installation from PyPI The third option is to install the package from The Python Package Index (PyPI), which may not be the latest version but still supports most of the features. The installation script is shown in Figure 7.

heading: A.2 Quick-start
text: We provide two ways for users to quickly get started with EasyInstruct. Users can either use the shell script or the Gradio app based on their specific needs.

heading: A.2.1 Shell Script
text: Step1: Prepare a configuration file. Users can easily configure the parameters of EasyInstruct in a YAML-style file or just quickly use the default parameters in the configuration files we provide. Figure 8 is an example of the configuration file for Self-Instruct.
Step2: Run the shell script. Users should first specify the configuration file and provide their own OpenAI API key. Then, run the following shell script in Figure 10 to launch the instruction generation or selection process.

heading: A.2.2 Gradio App
text: We provide a Gradio app for users to quickly get started with EasyInstruct. Users can choose to launch the Gradio App locally on their own machines or alternatively, they can also try the hosted Gradio App9 that we provide on HuggingFace Spaces.
9https://huggingface.co/spaces/zjunlp/ EasyInstruct.

heading: A.3 Detailed Hyper-Parameters
text: See Table 3.

heading: A.4 Prompt Template for Instruction Tuning
text: For both training and evaluation, we utilize the same prompt templates used by Alpaca-LoRA, shown in Table 4.

heading: A.5 API Services Available in EasyInstruct
text: Table 5 lists a range of API service providers and their corresponding LLM products that are currently available in EasyInstruct.
git clone https://github.com/zjunlp/EasyInstruct↪→ cd EasyInstruct pip install -e .
Figure 6: Installation script for local development.
pip install easyinstruct
Figure 7: Installation script using PyPI.
selector: source_file_path: target_dir: data/selections/ target_file_name: case.jsonl LengthSelector:
min_instruction_length: 3 max_instruction_length: 150 min_response_length: 1 max_response_length: 350
Deduplicator: RougeSelector:
threshold: 0.7 GPTScoreSelector:
engine: gpt-3.5-turbo threshold: 4
MTLDSelector: ttr_threshold: 0.72 min_mtld: 8 max_mtld: 22 PPLSelector: threshold: 200 model_name: gpt2 device: cuda RandomSelector: num_instructions_to_sample: 100 seed: 42
Figure 9: Example configuration file of Selectors.
config_file="" openai_api_key=""
python demo/run.py \ --config $config_file\ --openai_api_key $openai_api_key \
Figure 10: Shell script for quick-start of EasyInstruct.
A.6 Example features in the Prompts module

heading: A.7 Acknowledgements
text: We thank the developers of the self-instruct10 library for their significant contributions to the NLP community. We thank the LLaMA team for providing us access to the models, and open-source projects, including Alpaca11, Alpaca-LoRA12 and AlpacaEval13.
10https://github.com/yizhongw/self-instruct 11https://github.com/tatsu-lab/stanford_alpaca 12https://github.com/tloen/alpaca-lora 13https://github.com/tatsu-lab/alpaca_eval

heading: Future Work
text: Although our design choice is to decouple instruction processing and model training into two separate phases, we regard it as a limitation that may be addressed by future work. Statistics for evaluating efficiency. In our evaluation, we fine-tune a LLaMA2 (7B) model utilizing multiple modules implemented in EasyInstruct. Compared to models fine-tuned on other instruction datasets constructed without EasyInstruct, our model achieves optimal results, demonstrating EasyInstruct’s capability. Although we also qualitatively demonstrate the ease of writing code for instruction processing with multiple code samples and configuration files using EasyInstruct, a limitation is the lack of appropriate statistics for quantitatively evaluating efficiency