Package: py-data-modori
English |한국어Data-Modori: LMOps Tool for KoreanTEAMREBOOTT websiteHOTTeamAR platformTRY IT OUT📘Documentation|🛠️Installation|🤔Reporting IssuesData-Modori is a creative and advanced platform that guides you into the realm of data, offering diverse possibilities by collecting information from various sources. We gather all the puzzle pieces of data, assemble them into one, and invite you into the world of the information you desire.Data Integration: We collect data from various sources, integrating them into one central hub for your convenience.Flexible Analysis: Utilize advanced analysis tools to delve into your data, gaining new insights and perspectives.Customized Results: Organize and present data according to your requirements, delivering tailored results.User-Friendly Interface: An intuitive and easy-to-use interface allows users to harness the power of data without requiring advanced knowledge.Table of ContentsData-Modori: LMOps Tool for KoreanTable of ContentsInstallationData ProcessingData AnalysisData VisualizationBuild Up Config FilesDocumentationLicenseContributingAcknowledgementReferencesInstallationGet source code fromGithubgitclonehttps://github.com/teamreboott/data-modoricddata-modoriRun the following commands to install the latest basic data_juicer version in editable mode:pipinstall-v-e.Some OPs rely on some other too large or low-platform-compatibility third-party libraries. You can install optional dependencies as needed:pipinstall-v-e.# install a minimal dependencies, which support the basic functionspipinstall-v-e.[tools]# install a subset of tools dependenciesThe dependency options are listed below:TagDescription.or.[mini]Install minimal dependencies for basic Data-Modori..[all]Install all optional dependencies (including minimal dependencies and all of the following)..[sci]Install all dependencies for all OPs..[dist]Install dependencies for distributed data processing. (Experimental).[dev]Install dependencies for developing the package as contributors..[tools]Install dependencies for dedicated tools, such as quality classifiers.Data ProcessingRunprocess_data.pytool with your config as the argument to process your dataset.pythontools/process_data.py--configconfigs/process.yamlNote:For some operators that involve third-party models or resources which are not stored locally on your computer, it might be slow for the first running because these ops need to download corresponding resources into a directory first.
The default download cache directory is~/.cache/data_juicer. Change the cache location by setting the shell environment variable,DATA_JUICER_CACHE_HOMEto another directory, and you can also changeDATA_JUICER_MODELS_CACHEorDATA_JUICER_ASSETS_CACHEin the same way:# cache homeexportDATA_MODORI_CACHE_HOME="/path/to/another/directory"# cache modelsexportDATA_MODORI_MODELS_CACHE="/path/to/another/directory/models"# cache assetsexportDATA_MODORI_ASSETS_CACHE="/path/to/another/directory/assets"Data AnalysisRunanalyze_data.pytool with your config as the argument to analyse your dataset.pythontools/analyze_data.py--configconfigs/analyser.yamlNote:Analyser only compute stats of Filter ops. So extra Mapper or Deduplicator ops will be ignored in the analysis process.Data VisualizationRunapp.pytool to visualize your dataset in your browser.Note: only available for installation from source.streamlitrunapp.pyBuild Up Config FilesConfig files specify some global arguments, and an operator list for the
data process. You need to set:Global arguments: input/output dataset path, number of workers, etc.Operator list: list operators with their arguments used to process the dataset.You can build up your own config files by:➖：Modify from our example config fileconfig_all.yamlwhich includesallops and default
arguments. You just need toremoveops that you won't use and refine
some arguments of ops.➕：Build up your own config filesfrom scratch. You can refer our
example config fileconfig_all.yaml,op documents, and advancedBuild-Up Guide for developers.Besides the yaml files, you also have the flexibility to specify just
one (of several) parameters on the command line, which will override
the values in yaml files.pythonxxx.py--configconfigs/process.yaml--language_id_score_filter.lang=ko# Process config example for dataset# global parametersproject_name:'demo-process'dataset_path:'./data/test.json'# path to your dataset directory or fileexport_path:'./output/test.jsonl'np:4# number of subprocess to process your datasettext_keys:'content'# process schedule# a list of several process operators with their argumentsprocess:-language_id_score_filter:lang:'en'DocumentationOverviewOperatorsConfigsDeveloper GuideLicenseData-Modori is released under Apache License 2.0.ContributingWe are in a rapidly developing field and greatly welcome contributions of new
features, bug fixes and better documentations. Please refer toHow-to Guide for Developers.AcknowledgementData-Modori is used across various LLM products and research initiatives,
including industrial LLMs from Teamreboott AI TEAM(AR),
such as AUT for trade and AUW for work.We look forward to more of your experience, suggestions and discussions for collaboration!Data-Modori thanks and refers to several community projects, such asdata-juicer,Huggingface-Datasets,Bloom,Pile,Megatron-LM,DeepSpeed,Arrow,Ray,Beam,LM-Harness,HELM, ....ReferencesIf you find our work useful for your research or development, please kindly cite the followingpaper.@misc{chen2023datajuicer,
title={Data-Juicer: A One-Stop Data Processing System for Large Language Models},
author={Daoyuan Chen and Yilun Huang and Zhijian Ma and Hesen Chen and Xuchen Pan and Ce Ge and Dawei Gao and Yuexiang Xie and Zhaoyang Liu and Jinyang Gao and Yaliang Li and Bolin Ding and Jingren Zhou},
year={2023},
eprint={2309.02033},
archivePrefix={arXiv},
primaryClass={cs.LG}
}