Package: lm-datasets
lm-datasetslm-datasets is a collection of datasets for language model training including scripts for downloading, preprocesssing, and sampling.The documentation is availablehere.Quick startInstallationInstall thelm-datasetspackage withpip:pipinstalllm-datasetsIn order to keep the package minimal by default,lm-datasetscomes with optional dependencies useful for some use cases.
For example, if you want to have the text extraction for all available datasets, run:pipinstalllm-datasets[datasets]Download and text extractionTo download and extract the plain-text of one or more datasets, run the following command:lm_datasetsextract_text$DATASET_ID$OUTPUT_DIRBy default, output is saved as JSONL files. To change the output format, you can use the--output_formatargument as below:lm_datasetsextract_text$DATASET_ID$OUTPUT_DIR--output_formatparquet--output_compressionzstdAvailable datasetsA list or table with all available datasets can be print with the follow command:lm_datasetsprint_stats--print_outputmdToken count by languageLanguageTokensbg53 Bca5 Bcode250 Bcs128 Bda34 Bde795 Bel108 Ben6 Tes674 Bet15 Beu696 Mfi55 Bfr655 Bga767 Mgl70 Mhr8 Bhu179 Bit386 Blt24 Blv14 Bmt4 Bnl238 Bnn307 Mno9 Bpl223 Bpt187 Bro77 Bsh2 Msk47 Bsl11 Bsr10 Bsv89 Buk47 BToken count by sourceSourceTokensacademic_slovene_kas1 Bbgnc_admin_eur79 Mbgnc_news_corpus18 Mbrwac3 Bbulgarian_news283 Mbulnc567 Mcabernet712 Mcc_gigafida127 Mcolossal_oscar208 Bcroatian_news_engri695 Mcurlicat410 Mdanewsroom472 Mdanish_gigaword1 Bdewac2 Bdialogstudio0dk_clarin441 Menc20210estonian_reference_corpus175 Meurlex121 Beuscrawl423 Mga_bilingual_legistation4 Mga_universal_dependencies3 Mgreek_legal_code45 Mgreek_web_corpus3 Bhrwac1 Bitwac2 Bkorpus_malti366 Mlegal_mc429 Bmacocu23 Bmarcell_legislative_subcorpus_v231 Mnorwegian_cc5 Bopenlegaldata10 Boscar9 Toscar_opengptx245 Bparlamento_pt819 Mpes2o42 Bpl_nkjp1 Mpl_parliamentary_corpus671 Mproof_pile8 Bredpajama46 Bseimas_lt_en48 ksk_court_decisions11 Bsk_laws45 Mslwac_web1 Bsonar500 Msonar_new_media36 Mspanish_legal3 Bsrpkor0starcoder250 Bstate_related_latvian_web1 Mstyria_news409 Msv_gigaword1 Bsyn_v95 Buk_laws579 Mwiki12 Bwikibooks353 Mwikihow2 Mwikinews79 Mwikiquote268 Mwikisource2 Bwikivoyage132 Mylenews0Dataset viewerWe provide a Web-based application through streamlit to browse all datasets and their contained text content.
To start the app, first clone this repository, install dependencies, and run the following command:# clone is needed since streamlit does not support apps from modules yetgitclonehttps://github.com/malteos/lm-datasets.git

streamlitrunsrc/lm_datasets/viewer/app.py--\--raw_datasets_dir=$RAW_DATASETS_DIR\--output_dir=$PROCESSED_DATASET_DIRDevelopment & ContributionsSetup environmentTo setup, your local development environment we recommend conda and cloning the repository.
The repository also includes settings and launch scripts for VSCode.gitclonegit@github.com:malteos/lm-datasets.gitcdlm-datasets

condacreate-nlm-datasetspython=3.10
condaactivatelm-datasets

pipinstall-rrequirements.txtAlternatively, you can install the Python package directly from the dev branch:pipinstallgit+https://github.com/malteos/lm-datasets.git@devInstall the pre-commit hooksThis repository uses git hooks to validate code quality and formatting.pre-commit install
git config --bool flake8.strict true  # Makes the commit fail if flake8 reports an errorTo run the hooks:pre-commit run --all-filesTestingThe tests can be executed with:pytest --doctest-modules --cov-report term --cov=lm_datasetsAcknowledgementsThe work on the lm-datasets software is partially funded by theGerman Federal Ministry for Economic Affairs and Climate Action (BMWK)through the projectOpenGPT-X(project no. 68GX21007D).LicenseApache 2.0(Please note that the actual datasets are released with different licenses)