Package: py-data-juicer
English |中文Data-Juicer:  A One-Stop Data Processing System for Large Language ModelsData-Juicer is a one-stop data processing system to make data higher-quality,
juicier, and more digestible for LLMs.
This project is being actively updated and maintained, and we will periodically enhance and add more features and data recipes. We welcome you to join us in promoting LLM data development and research!If you find Data-Juicer useful for your research or development, please kindly
cite ourwork.Welcome to join ourSlack channel,DingDing group, or WeChat group (scan the QR code below with WeChat) for discussion.News[2024-01-05] We releaseData-Juicer v0.1.3now!
In this new version, we supportmore Python versions(3.7-3.10), and supportmultimodaldatasetconverting/processing(Including texts, images, and audios. More modalities will be supported in the future).
Besides, our paper is also updated tov3.[2023-10-13] Our first data-centric LLM competition begins! Please
visit the competition's official websites, FT-Data Ranker (1B Track,7B Track), for more information.[2023-10-8] We update our paper to the 2nd version and release the corresponding version 0.1.2 of Data-Juicer!Table of ContentsData-Juicer: A One-Stop Data Processing System for Large Language ModelsTable of ContentsFeaturesPrerequisitesInstallationFrom SourceUsing pipUsing DockerInstallation checkQuick StartData ProcessingData AnalysisData VisualizationBuild Up Config FilesPreprocess raw data (Optional)For Docker UsersDocumentation | 文档Data RecipesDemosLicenseContributingAcknowledgementReferencesFeaturesSystematic & Reusable:
Empowering users with a systematic library of 20+ reusableconfig recipes, 50+ coreOPs, and feature-rich
dedicatedtoolkits, designed to
function independently of specific LLM datasets and processing pipelines.Data-in-the-loop: Allowing detailed data analyses with an automated
report generation feature for a deeper understanding of your dataset. Coupled with multi-dimension automatic evaluation capabilities, it supports a timely feedback loop at multiple stages in the LLM development process.Comprehensive Data Processing Recipes: Offering tens ofpre-built data
processing recipesfor pre-training, fine-tuning, en, zh, and more scenarios. Validated on
reference LLaMA models.Enhanced Efficiency: Providing a speedy data processing pipeline
requiring less memory and CPU usage, optimized for maximum productivity.Flexible & Extensible: Accommodating most types of data formats (e.g., jsonl, parquet, csv, ...) and allowing flexible combinations of OPs. Feel free toimplement your own OPsfor customizable data processing.User-Friendly Experience: Designed for simplicity, withcomprehensive documentation,easy start guidesanddemo configs, and intuitive configuration with simple adding/removing OPs fromexisting configs.PrerequisitesRecommend Python>=3.7,<=3.10gcc >= 5 (at least C++14 support)InstallationFrom SourceRun the following commands to install the latest basicdata_juicerversion in
editable mode:cd<path_to_data_juicer>
pipinstall-v-e.Some OPs rely on some other too large or low-platform-compatibility third-party libraries. You can install optional dependencies as needed:cd<path_to_data_juicer>
pipinstall-v-e.# install a minimal dependencies, which support the basic functionspipinstall-v-e.[tools]# install a subset of tools dependenciesThe dependency options are listed below:TagDescription.or.[mini]Install minimal dependencies for basic Data-Juicer..[all]Install all optional dependencies (including minimal dependencies and all of the following)..[sci]Install all dependencies for all OPs..[dist]Install dependencies for distributed data processing. (Experimental).[dev]Install dependencies for developing the package as contributors..[tools]Install dependencies for dedicated tools, such as quality classifiers.Using pipRun the following command to install the latest releaseddata_juicerusingpip:pipinstallpy-data-juicerNote:only the basic APIs indata_juicerand two basic tools
(dataprocessingandanalysis) are available in this way. If you want customizable
and complete functions, we recommend you installdata_juicerfrom source.The release versions from pypi have a certain lag compared to the latest version from source.
So if you want to follow the latest functions ofdata_juicer, we recommend you installfrom source.Using DockerYou caneither pull our pre-built image from DockerHub:dockerpulldatajuicer/data-juicer:<version_tag>or run the following command to build the docker image including the
latestdata-juicerwith providedDockerfile:dockerbuild-tdata-juicer:<version_tag>.Installation checkimportdata_juicerasdjprint(dj.__version__)Quick StartData ProcessingRunprocess_data.pytool ordj-processcommand line tool with your config as the argument to process
your dataset.# only for installation from sourcepythontools/process_data.py--configconfigs/demo/process.yaml# use command line tooldj-process--configconfigs/demo/process.yamlNote:For some operators that involve third-party models or resources which are not stored locally on your computer, it might be slow for the first running because these ops need to download corresponding resources into a directory first.
The default download cache directory is~/.cache/data_juicer. Change the cache location by setting the shell environment variable,DATA_JUICER_CACHE_HOMEto another directory, and you can also changeDATA_JUICER_MODELS_CACHEorDATA_JUICER_ASSETS_CACHEin the same way:# cache homeexportDATA_JUICER_CACHE_HOME="/path/to/another/directory"# cache modelsexportDATA_JUICER_MODELS_CACHE="/path/to/another/directory/models"# cache assetsexportDATA_JUICER_ASSETS_CACHE="/path/to/another/directory/assets"Data AnalysisRunanalyze_data.pytool ordj-analyzecommand line tool with your config as the argument to analyse your dataset.# only for installation from sourcepythontools/analyze_data.py--configconfigs/demo/analyser.yaml# use command line tooldj-analyze--configconfigs/demo/analyser.yamlNote:Analyser only compute stats of Filter ops. So extra Mapper or Deduplicator ops will be ignored in the analysis process.Data VisualizationRunapp.pytool to visualize your dataset in your browser.Note: only available for installation from source.streamlitrunapp.pyBuild Up Config FilesConfig files specify some global arguments, and an operator list for the
data process. You need to set:Global arguments: input/output dataset path, number of workers, etc.Operator list: list operators with their arguments used to process the dataset.You can build up your own config files by:➖：Modify from our example config fileconfig_all.yamlwhich includesallops and default
arguments. You just need toremoveops that you won't use and refine
some arguments of ops.➕：Build up your own config filesfrom scratch. You can refer our
example config fileconfig_all.yaml,op documents, and advancedBuild-Up Guide for developers.Besides the yaml files, you also have the flexibility to specify just
one (of several) parameters on the command line, which will override
the values in yaml files.pythonxxx.py--configconfigs/demo/process.yaml--language_id_score_filter.lang=enThe basic config format and definition is shown below.Preprocess Raw Data (Optional)Our formatters support some common input dataset formats for now:Multi-sample in one file: jsonl/json, parquet, csv/tsv, etc.Single-sample in one file: txt, code, docx, pdf, etc.However, data from different sources are complicated and diverse. Such as:Raw arXiv data downloaded from S3include thousands of tar files and even more gzip files in them, and expected tex files are embedded in the gzip files so they are hard to obtain directly.Some crawled data include different kinds of files (pdf, html, docx, etc.). And extra information like tables, charts, and so on is hard to extract.It's impossible to handle all kinds of data in Data-Juicer, issues/PRs are welcome to contribute to process new data types!Thus, we provide somecommon preprocessing toolsintools/preprocessfor you to preprocess these data.You are welcome to make your contributions to new preprocessing tools for the community.Wehighly recommendthat complicated data can be preprocessed to jsonl or parquet files.For Docker UsersIf you build or pull the docker image ofdata-juicer, you can run the commands or tools mentioned above using this docker image.Run directly:# run the data processing directlydockerrun--rm\# remove container after the processing--namedj\# name of the container-v<host_data_path>:<image_data_path>\# mount data or config directory into the container-v~/.cache/:/root/.cache/\# mount the cache directory into the container to reuse caches and models (recommended)data-juicer:<version_tag>\# image to rundj-process--config/path/to/config.yaml# similar data processing commandsOr enter into the running container and run commands in editable mode:# start the containerdockerrun-dit\# run the container in the background--rm\--namedj\-v<host_data_path>:<image_data_path>\-v~/.cache/:/root/.cache/\data-juicer:latest/bin/bash# enter into this container and then you can use data-juicer in editable modedockerexec-it<container_id>bashDocumentation | 文档Overview|概览Operator Zoo|算子库Configs|配置系统Developer Guide|开发者指南Dedicated Toolkits | 专用工具箱Quality Classifier|质量分类器Auto Evaluation|自动评测Preprocess|前处理Postprocess|后处理Third-parties (LLM Ecosystems)|第三方库（大语言模型生态）API referencesData RecipesRecipes for data process in BLOOMRecipes for data process in RedPajamaRefined recipes for pre-training dataRefined recipes for fine-tuning dataDemosIntroduction to Data-Juicer [ModelScope] [HuggingFace]Data Visualization:Basic Statistics [ModelScope] [HuggingFace]Lexical Diversity [ModelScope] [HuggingFace]Operator Effect [ModelScope] [HuggingFace]Data Processing:Scientific Literature (e.g.arXiv) [ModelScope] [HuggingFace]Programming Code (e.g.TheStack) [ModelScope] [HuggingFace]Chinese Instruction Data (e.g.Alpaca-CoT) [ModelScope] [HuggingFace]Tool Pool:Dataset Splitting by Language [ModelScope] [HuggingFace]Quality Classifier for CommonCrawl [ModelScope] [HuggingFace]Auto Evaluation onHELM[ModelScope] [HuggingFace]Data Sampling and Mixture [ModelScope] [HuggingFace]Data Processing Loop [ModelScope] [HuggingFace]Data Processing HPO [ModelScope] [HuggingFace]LicenseData-Juicer is released under Apache License 2.0.ContributingWe are in a rapidly developing field and greatly welcome contributions of new
features, bug fixes and better documentations. Please refer toHow-to Guide for Developers.If you have any questions, please join ourdiscussion groups.AcknowledgementData-Juicer is used across various LLM products and research initiatives,
including industrial LLMs from Alibaba Cloud's Tongyi, such as Dianjin for
financial analysis, and Zhiwen for reading assistant, as well as the Alibaba
Cloud's platform for AI (PAI).
We look forward to more of your experience, suggestions and discussions for collaboration!Data-Juicer thanks and refers to several community projects, such asHuggingface-Datasets,Bloom,RedPajama,Pile,Alpaca-Cot,Megatron-LM,DeepSpeed,Arrow,Ray,Beam,LM-Harness,HELM, ....ReferencesIf you find our work useful for your research or development, please kindly cite the followingpaper.@misc{chen2023datajuicer,
title={Data-Juicer: A One-Stop Data Processing System for Large Language Models},
author={Daoyuan Chen and Yilun Huang and Zhijian Ma and Hesen Chen and Xuchen Pan and Ce Ge and Dawei Gao and Yuexiang Xie and Zhaoyang Liu and Jinyang Gao and Yaliang Li and Bolin Ding and Jingren Zhou},
year={2023},
eprint={2309.02033},
archivePrefix={arXiv},
primaryClass={cs.LG}
}