Package: fabricator-ai
A flexible open-source framework to generate datasets with large language models.Installation|Basic Concepts|Examples|Tutorials|Paper|CitationNews[10/23]We released the first version of this repository on PyPI. You can install it viapip install fabricator-ai.[10/23]Our paper got accepted at EMNLP 2023. You can find the preprinthere. You can find the experimental scripts under release v0.1.0.[09/23]Support forgpt-3.5-turbo-instructadded in the newHaystackrelease![08/23]Added several experimental scripts to investigate the generation and annotation ability ofgpt-3.5-turboon various downstream tasks + the influence of few-shot examples on the performance for different downstream tasks.[07/23]Refactorings of majors classes - you can now simply use our BasePrompt class to create your own customized prompts for every downstream task![07/23]Added dataset transformations for token classification to prompt LLMs with textual spans rather than with list of tags.[06/23]Initial version of fabricator supporting text classification and question answering tasks.OverviewThis repository:isan easy-to-use open-source libraryto generate datasets with large language models. If you want to train
a model on a specific domain / label distribution / downstream task, you can use this framework to generate
a dataset for it.builds on top of deepset's haystack and huggingface's datasetslibraries. Thus, we support a wide range
of language models and you can load and use the generated datasets as you know it from the Datasets library for your
model training.ishighly flexibleand offers various adaptions possibilities such as
prompt customization, integration and sampling of fewshot examples or annotation of the unlabeled datasets.InstallationUsing conda:git clone git@github.com:flairNLP/fabricator.git
cd fabricator
conda create -y -n fabricator python=3.10
conda activate fabricator
pip install fabricator-aiIf you want to install in editable mode, you can use the following command:pip install -e .Basic ConceptsThis framework is based on the idea of using large language models to generate datasets for specific tasks. To do so,
we need four basic modules: a dataset, a prompt, a language model and a generator:Dataset: We usehuggingface's datasets libraryto load fewshot or
unlabeled datasets and store the generated or annotated datasets with theirDatasetclass. Once
created, you can share the dataset with others via the hub or use it for your model training.Prompt: A prompt is the instruction made to the language model. It can be a simple sentence or a more complex
template with placeholders. We provide an easy interface for custom dataset generation prompts in which you can specify
label options for the LLM to choose from, provide fewshot examples to support the prompt with or annotate an unlabeled
dataset in a specific way.LLM: We usedeepset's haystack libraryas our LLM interface. deepset
supports a wide range of LLMs including OpenAI, all models from the HuggingFace model hub and many more.Generator: The generator is the core of this framework. It takes a dataset, a prompt and a LLM and generates a
dataset based on your specifications.ExamplesWith our library, you can generate datasets for any task you want. You can start as simple
as that:Generate a dataset from scratchimportosfromhaystack.nodesimportPromptNodefromfabricatorimportDatasetGeneratorfromfabricator.promptsimportBasePromptprompt=BasePrompt(task_description="Generate a short movie review.",)prompt_node=PromptNode(model_name_or_path="gpt-3.5-turbo",api_key=os.environ.get("OPENAI_API_KEY"),max_length=100,)generator=DatasetGenerator(prompt_node)generated_dataset=generator.generate(prompt_template=prompt,max_prompt_calls=10,)generated_dataset.push_to_hub("your-first-generated-dataset")In our tutorial, we introduce how to create classification datasets with label options to choose from, how to include
fewshot examples or how to annotate unlabeled data into predefined categories.CitationIf you find this repository useful, please cite our work.@misc{golde2023fabricator,
      title={Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs}, 
      author={Jonas Golde and Patrick Haller and Felix Hamborg and Julian Risch and Alan Akbik},
      year={2023},
      eprint={2309.09582},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}