Package: text-machina
Unifying strategies to build MGT datasets in a single frameworkTextMachina is a modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as:🔎Detection: detect whether a text has been generated by an LLM.🕵️‍♂️Attribution: identify what LLM has generated a text.🚧Boundary detection: find the boundary between human and generated text.🎨Mixcase: ascertain whether specific text spans are human-written or generated by LLMs.TextMachina provides a user-friendly pipeline that abstracts away the inherent intricacies of building MGT datasets:🦜LLM integrations: easily integrates any LLM provider. Currently,supports LLMs from Anthropic, Cohere, OpenAI, Google Vertex AI, Amazon Bedrock, AI21, Azure OpenAI, models deployed on VLLM and TRT inference servers, and any model from HuggingFace deployed either locally or remotely through Inference API or Inference Endpoints. Seemodelsto implement your own LLM provider.✍️Prompt templating: just write your prompt template with placeholders and letextractors to fill the template and prepare a prompt for an LLM. Seeextractorsto implement your own extractors and learn more about the placeholders for each extractor.🔒Constrained decoding: automatically infer LLM decoding hyper-parameters from the human texts to improve the quality and reduce the biases of your MGT datasets. Seeconstrainersto implement your own constrainers.🛠️Post-processing: post-process functions aimed to improve the quality of any MGT dataset and prevent common biases and artifacts. Seepostprocessingto add new postprocess functions.🌈Bias mitigation:is built with bias prevention in mind and helps you across all the pipeline to prevent introducing spurious correlations in your datasets.📊Dataset exploration: explore the generated datasets and quantify its quality with a set of metrics. Seemetricsandinteractiveto implement your own metrics and visualizations.The following diagram depicts the's pipeline.🔧 InstallationYou can install all the dependencies with pip:pip install text-machina[all]or just with specific dependencies for an specific LLM provider or development dependencies (seesetup.py):pip install text-machina[anthropic,dev]You can also install directly from source:pip install .[all]If you're planning to modify the code for specific use cases, you can installin development mode:pip install -e .[dev]👀 Quick TourOnce installed, you are ready to usefor building MGT datasets either using theCLIor programmatically.📟 Using the CLIThe first step is to define a YAML configuration file or a directory tree containing YAML files. Read theexamples/learningfiles to learn how to define configuration using different providers and extractors for different tasks. Take a look toexamples/use_casesto see configurations for specific use cases.Then, we can call theexploreandgenerateendpoints of's CLI. Theexploreendpoint allows to inspect a small generated dataset using an specific configuration through an interactive interface. For instance, let's suppose we want to check how an MGT detection dataset generated usingXSumnews articles andgpt-3.5-turbo-instructlooks like, and compute some metrics:text-machinaexplore--config-pathetc/examples/xsum_gpt-3-5-turbo-instruct_openai.yaml\--task-typedetection\--metrics-pathetc/metrics.yaml\--max-generations10Great! Our dataset seems to look great, no artifacts, no biases, and high-quality text using this configuration. Let's now generate a whole dataset for MGT detection using that config file. Thegenerateendpoint allows you to do that:text-machinagenerate--config-pathetc/examples/xsum_gpt-3-5-turbo-instruct_openai.yaml\--task-typedetectionA run name will be assigned to your execution andwill cache results behind the scenes. If your run is interrupted at any point, you can use--run-name <run-name>to recover the progress and continue generating your dataset.👩‍💻 ProgrammaticallyYou can also useprogrammatically. To do that, instantiate a dataset generator by callingget_generatorwith aConfigobject, and run itsgeneratemethod. TheConfigobject must contain the input, model, and generation configs, together with the task type for which the MGT dataset will be generated. Let's replicate the previous experiment programmatically:fromtext_machinaimportget_generatorfromtext_machinaimportConfig,InputConfig,ModelConfiginput_config=InputConfig(domain="news",language="en",quantity=10,random_sample_human=True,dataset="xsum",dataset_text_column="document",dataset_params={"split":"test"},template=("Write a news article whose summary is '{summary}'""using the entities:{entities}\n\nArticle:"),extractor="combined",extractors_list=["auxiliary.Auxiliary","entity_list.EntityList"],max_input_tokens=256,)model_config=ModelConfig(provider="openai",model_name="gpt-3.5-turbo-instruct",api_type="COMPLETION",threads=8,max_retries=5,timeout=20,)generation_config={"temperature":0.7,"presence_penalty":1.0}config=Config(input=input_config,model=model_config,generation=generation_config,task_type="detection",)generator=get_generator(config)dataset=generator.generate()🛠️ Supported taskscan generate datasets for MGT detection, attribution, boundary detection, and mixcase detection:Example from a detection task.Example from an attribution task.Example from a boundary detection task.Example from a mixcase task (tagging), interleaving generated sentences with human texts.Example from a mixcase task (tagging), interleaving generated word spans with human texts.However, the users can build datasets for other tasks not included injust by leveraging the provided task types. For instance, datasets for mixcase classification can be built from datasets for mixcase tagging, or datasets for mixcase attribution can be built using the generation model name as label.🔄 Common Use CasesThere is a set of common use cases with. Here's how to carry them out using theexploreandgenerateendpoints.Use caseCommandExplore a dataset of 10 samples for MGT detection and show metricstext-machina explore \--config-path config.yaml \--task-type detection \--max-generations 10 \--metrics_path metrics.yamlExplore an existing dataset for MGT detection and show metricstext-machina explore \--config-path config.yaml \--run-name greedy-bear \--task-type detection \--metrics_path metrics.yamlGenerate a dataset for MGT detectiontext-machina generate \--config-path config.yaml \--task-type detectionGenerate a dataset for MGT attributiontext-machina generate \--config-path config.yaml \--task-type attributionGenerate a dataset for boundary detectiontext-machina generate \--config-path config.yaml \--task-type boundaryGenerate a dataset for mixcase detectiontext-machina generate \--config-path config.yaml \--task-type mixcaseGenerate a dataset for MGT detection using config files in a directory treetext-machina generate \--config-path configs/ \--task-type detection💾 CachingTextMachina caches each dataset it generates through the CLI endpoints under a run name.
The specific run name is given as the last message in the logs, and can be used with--run-name <run-name>to continue from interrupted runs.
The default cache dir used byTextMachina is/tmp/text_machina_cache.
It can be modified by settingTEXT_MACHINA_CACHE_DIRto a different path.⚠️ Notes and LimitationsAlthough you can use any kind of extractor to build boundary detection datasets, it is highly recommended to use thesentence_prefixorword_prefixextractors with a random number of sentences/words to avoid biases that lead boundary detection models to just count sentences or words.attempts to remove disclosure patterns (e.g., "As an AI language model ...") with a limited set of regular expressions, but they depend on the LLM and the language. We strictly recommend to firstexploreyour dataset looking for these biases, and modify the postprocessing or the prompt template accordingly to remove them.Generating multilingual datasets is not well supported yet. At this moment, we recommend to generate independent datasets for each language and combine them together out of.Generating machine-generated code datasets is not well supported yet.📖 Citation@misc{sarvazyan2024textmachina,
      title={TextMachina: Seamless Generation of Machine-Generated Text Datasets}, 
      author={Areg Mikael Sarvazyan and José Ángel González and Marc Franco-Salvador},
      year={2024},
      eprint={2401.03946},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}🤝 ContributeFeel free to contribute toby raising an issue.Please install and use thedev-toolsfor correctly formatting the code when contributing to this repo.🏭 Commercial PurposesPlease, contactstuart.winter-tear@genaios.aiandmarc.franco@genaios.aiif you are interested in using TextMachina for commercial purposes.