Package: easyinstruct
An Easy-to-use Instruction Processing Framework for Large Language Models.Project•Paper•Demo•Overview•Installation•Quickstart•How To Use•Docs•Citation•ContributorsTable of ContentsWhat's NewOverviewInstallationQuickstartShell ScriptGradio AppUse EasyInstructGeneratorsSelectorsPromptsEnginesCitationContributors🔔News2024-2-6 We release the paper "EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models".2024-2-5 We release version 0.1.2, supporting for new features and optimising the function interface.2023-12-9 The paper "When Do Program-of-Thoughts Work for Reasoning?" (supported by EasyInstruct), is accepted by AAAI 2024!2023-10-28 We release version 0.1.1, supporting for new features of instruction generation and instruction selection.2023-8-9 We release version 0.0.6, supporting Cohere API calls.2023-7-12 We releaseEasyEdit, an easy-to-use framework to edit Large Language Models.Previous news2023-5-23 We release version 0.0.5, removing requirement of llama-cpp-python.2023-5-16 We release version 0.0.4, fixing some problems.2023-4-21 We release version 0.0.3, check out ourdocumentationsfor more details.2023-3-25 We release version 0.0.2, suporting IndexPrompt, MMPrompt, IEPrompt and more LLMs2023-3-13 We release version 0.0.1, supporting in-context learning, chain-of-thought with ChatGPT.This repository is a subproject ofKnowLM.🌟OverviewEasyInstruct is a Python package which is proposed as an easy-to-use instruction processing framework for Large Language Models(LLMs) like GPT-4, LLaMA, ChatGLM in your research experiments. EasyInstruct modularizes instruction generation, selection, and prompting, while also considering their combination and interaction.The current supported instruction generation techniques are as follows:MethodsDescriptionSelf-InstructThe method that randomly samples a few instructions from a human-annotated seed tasks pool as demonstrations and prompts an LLM to generate more instructions and corresponding input-output pairs.Evol-InstructThe method that incrementally upgrades an initial set of instructions into more complex instructions by prompting an LLM with specific prompts.BacktranslationThe method that creates an instruction following training instance by predicting an instruction that would be correctly answered by a portion of a document of the corpus.KG2InstructThe method that creates an instruction following training instance by predicting an instruction that would be correctly answered by a portion of a document of the corpus.The current supported instruction selection metrics are as follows:MetricsNotationDescriptionLength$Len$The bounded length of every pair of instruction and response.Perplexity$PPL$The exponentiated average negative log-likelihood of response.MTLD$MTLD$Measure of textual lexical diversity, the mean length of sequential words in a text that maintains a minimum threshold TTR score.ROUGE$ROUGE$Recall-Oriented Understudy for Gisting Evaluation, a set of metrics used for evaluating similarities between sentences.GPT score$GPT$The score of whether the output is a good example of how AI Assistant should respond to the user's instruction, provided by ChatGPT.CIRS$CIRS$The score using the abstract syntax tree to encode structural and logical attributes, to measure the correlation between code and reasoning abilities.API service providers and their corresponding LLM products that are currently available:ModelDescriptionDefault VersionOpenAIGPT-3.5A set of models that improve on GPT-3 and can understand as well as generate natural language or code.gpt-3.5-turboGPT-4A set of models that improve on GPT-3.5 and can understand as well as generate natural language or code.gpt-4AnthropicClaudeA next-generation AI assistant based on Anthropic’s research into training helpful, honest, and harmless AI systems.claude-2.0Claude-InstantA lighter, less expensive, and much faster option than Claude.claude-instant-1.2CohereCommandA flagship text generation model of Cohere trained to follow user commands and to be instantly useful in practical business applications.commandCommand-LightA light version of Command models that are faster but may produce lower-quality generated text.command-light🔧InstallationInstallation from git repo branch:pip install git+https://github.com/zjunlp/EasyInstruct@mainInstallation for local development:git clone https://github.com/zjunlp/EasyInstruct
cd EasyInstruct
pip install -e .Installation using PyPI (not the latest version):pip install easyinstruct -i https://pypi.org/simple⏩QuickstartWe provide two ways for users to quickly get started with EasyInstruct. You can either use the shell script or the Gradio app based on your specific needs.Shell ScriptStep1: Prepare a configuration fileUsers can easily configure the parameters of EasyInstruct in a YAML-style file or just quickly use the default parameters in the configuration files we provide. Following is an example of the configuration file for Self-Instruct:generator:SelfInstructGenerator:target_dir:data/generations/data_format:alpacaseed_tasks_path:data/seed_tasks.jsonlgenerated_instructions_path:generated_instructions.jsonlgenerated_instances_path:generated_instances.jsonlnum_instructions_to_generate:100engine:gpt-3.5-turbonum_prompt_instructions:8More example configuration files can be found atconfigs.Step2: Run the shell scriptUsers should first specify the configuration file and provide their own OpenAI API key. Then, run the following shell script to launch the instruction generation or selection process.config_file=""openai_api_key=""pythondemo/run.py\--config$config_file\--openai_api_key$openai_api_key\Gradio AppWe provide a Gradio app for users to quickly get started with EasyInstruct. You can run the following command to launch the Gradio app locally on the port7860(if available).pythondemo/app.pyWe also host a running gradio app in HuggingFace Spaces. You can try it outhere.📌Use EasyInstructPlease refer to ourdocumentationsfor more details.GeneratorsTheGeneratorsmodule streamlines the process of instruction data generation, allowing for the generation of instruction data based on seed data. You can choose the appropriate generator based on your specific needs.BaseGeneratorBaseGeneratoris the base class for all generators.You can also easily inherit this base class to customize your own generator class. Just override the__init__andgeneratemethod.SelfInstructGeneratorSelfInstructGeneratoris the class for the instruction generation method of Self-Instruct. SeeSelf-Instruct: Aligning Language Model with Self Generated Instructionsfor more details.ExamplefromeasyinstructimportSelfInstructGeneratorfromeasyinstruct.utils.apiimportset_openai_key# Step1: Set your own API-KEYset_openai_key("YOUR-KEY")# Step2: Declare a generator classgenerator=SelfInstructGenerator(num_instructions_to_generate=10)# Step3: Generate self-instruct datagenerator.generate()BacktranslationGeneratorBacktranslationGeneratoris the class for the instruction generation method of Instruction Backtranslation. SeeSelf-Alignment with Instruction Backtranslationfor more details.ExamplefromeasyinstructimportBacktranslationGeneratorfromeasyinstruct.utils.apiimportset_openai_key# Step1: Set your own API-KEYset_openai_key("YOUR-KEY")# Step2: Declare a generator classgenerator=BacktranslationGenerator(num_instructions_to_generate=10)# Step3: Generate backtranslation datagenerator.generate()EvolInstructGeneratorEvolInstructGeneratoris the class for the instruction generation method of EvolInstruct. SeeWizardLM: Empowering Large Language Models to Follow Complex Instructionsfor more details.ExamplefromeasyinstructimportEvolInstructGeneratorfromeasyinstruct.utils.apiimportset_openai_key# Step1: Set your own API-KEYset_openai_key("YOUR-KEY")# Step2: Declare a generator classgenerator=EvolInstructGenerator(num_instructions_to_generate=10)# Step3: Generate evolution datagenerator.generate()KG2InstructGeneratorKG2InstructGeneratoris the class for the instruction generation method of KG2Instruct. SeeInstructIE: A Chinese Instruction-based Information Extraction Datasetfor more details.SelectorsTheSelectorsmodule standardizes the instruction selection process, enabling the extraction of high-quality instruction datasets from raw, unprocessed instruction data. The raw data can be sourced from publicly available instruction datasets or generated by the framework itself. You can choose the appropriate selector based on your specific needs.BaseSelectorBaseSelectoris the base class for all selectors.You can also easily inherit this base class to customize your own selector class. Just override the__init__and__process__method.DeduplicatorDeduplicatoris the class for eliminating duplicate instruction samples that could adversely affect both pre-training stability and the performance of LLMs.Deduplicatorcan also enables efficient use and optimization of storage space.LengthSelectorLengthSelectoris the class for selecting instruction samples based on the length of the instruction. Instructions that are too long or too short can affect data quality and are not conducive to instruction tuning.RougeSelectorRougeSelectoris the class for selecting instruction samples based on the ROUGE metric which is often used for evaluating the quality of automated generation of text.GPTScoreSelectorGPTScoreSelectoris the class for selecting instruction samples based on the GPT score, which reflects whether the output is a good example of how AI Assistant should respond to the user's instruction, provided by ChatGPT.PPLSelectorPPLSelectoris the class for selecting instruction samples based on the perplexity, which is the exponentiated average negative log-likelihood of response.MTLDSelectorMTLDSelectoris the class for selecting instruction samples based on the MTLD, which is short for Measure of Textual Lexical Diversity.CodeSelectorCodeSelectoris the class for selecting code instruction samples based on the Complexity-Impacted Reasoning Score (CIRS), which combines structural and logical attributes, to measure the correlation between code and reasoning abilities. SeeWhen Do Program-of-Thoughts Work for Reasoning?for more details.ExamplefromeasyinstructimportCodeSelector# Step1: Specify your source file of code instructionssrc_file="data/code_example.json"# Step2: Declare a code selecter classselector=CodeSelector(source_file_path=src_file,target_dir="data/selections/",manually_partion_data=True,min_boundary=0.125,max_boundary=0.5,automatically_partion_data=True,k_means_cluster_number=2,)# Step3: Process the code instructionsselector.process()MultiSelectorMultiSelectoris the class for combining multiple appropricate selectors based on your specific needs.PromptsThePromptsmodule standardizes the instruction prompting step, where user requests are constructed as instruction prompts and sent to specific LLMs to obtain responses. You can choose the appropriate prompting method based on your specific needs.Please check outlinkfor more detials.EnginesTheEnginesmodule standardizes the instruction execution process, enabling the execution of instruction prompts on specific locally deployed LLMs. You can choose the appropriate engine based on your specific needs.Please check outlinkfor more detials.🚩CitationPlease cite our repository if you use EasyInstruct in your work.@misc{easyinstruct,author={Yixin Ou and Ningyu Zhang and Honghao Gui and Ziwen Xu and Shuofei Qiao and Zhen Bi and Huajun Chen},title={EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models},year={2023},url={https://github.com/zjunlp/EasyInstruct},}@misc{knowlm,author={Ningyu Zhang and Jintian Zhang and Xiaohan Wang and Honghao Gui and Kangwei Liu and Yinuo Jiang and Xiang Chen and Shengyu Mao and Shuofei Qiao and Yuqi Zhu and Zhen Bi and Jing Chen and Xiaozhuan Liang and Yixin Ou and Runnan Fang and Zekun Xi and Xin Xu and Lei Li and Peng Wang and Mengru Wang and Yunzhi Yao and Bozhong Tian and Yin Fang and Guozhou Zheng and Huajun Chen},title={KnowLM: An Open-sourced Knowledgeable Large Langugae Model Framework},year={2023},url={http://knowlm.zjukg.cn/},}@misc{bi2023programofthoughts,author={Zhen Bi and Ningyu Zhang and Yinuo Jiang and Shumin Deng and Guozhou Zheng and Huajun Chen},title={When Do Program-of-Thoughts Work for Reasoning?},year={2023},eprint={2308.15452},archivePrefix={arXiv},primaryClass={cs.CL}}🎉ContributorsWe will offer long-term maintenance to fix bugs, solve issues and meet new requests. So if you have any problems, please put issues to us.Other Related ProjectsSelf-InstructAlpaca🙌 We would like to express our heartfelt gratitude for the contribution ofSelf-Instructto our project, as we have utilized portions of their source code in our project.