Package: langumo
langumoThe unified corpus building environment for Language Models.Table of contentsIntroductionMain featuresDependenciesInstallationWith pipFrom sourceQuick start guideUsageDocumentationLicenseIntroductionlangumois anunified corpus building environment for Language Models.langumoprovides pipelines for building text-based datasets. Constructing
datasets requires complicated pipelines (e.g. parsing, shuffling and
tokenization). Moreover, if corpora are collected from different sources, it
would be a problem to extract data from various formats.langumohelps to
build a dataset with the diverse formats simply at once.Main featuresEasy to build, simple to add new corpus format.Fast building through performance optimizations (even written in Python).Supports multi-processing in parsing corpora.Extremely less memory usage.All-in-one environment. Never mind the internal procedures!Does not need to write codes for new corpus. Instead, add to the build
configuration simply.Dependenciesnltkcoloramapyyaml>=5.3.1tqdm>=4.46.0tokenizers==0.8.1mwparserfromhell>=0.5.4kss==1.3.1InstallationWith piplangumocan be installed usingpipas follows:$pipinstalllangumoFrom sourceYou can installlangumofrom source by cloning the repository and running:$gitclonehttps://github.com/affjljoo3581/langumo.git
$cdlangumo
$pythonsetup.pyinstallQuick start guideLet's build aWikipediadataset. First, installlangumoin your virtual
enviornment.$pipinstalllangumoAfter installinglangumo, create a workspace to use in build.$mkdirworkspace
$cdworkspaceBefore creating the dataset, we need aWikipedia dump file(which is a source of the dataset). You can get various
versions of Wikipedia dump files fromhere.
In this tutorial, we will usea part of Wikipedia dump file.
Download the file with your browser and move toworkspace/src. Or, usewgetto get the file in terminal simply:$wget-Psrchttps://dumps.wikimedia.org/enwiki/20200901/enwiki-20200901-pages-articles1.xml-p1p30303.bz2langumoneeds a build configuration file which contains the details of
dataset. Createbuild.ymlfile toworkspaceand write belows:langumo:inputs:-path:src/enwiki-20200901-pages-articles1.xml-p1p30303.bz2parser:langumo.parsers.WikipediaParserbuild:parsing:num-workers:8# The number of CPU cores you have.tokenization:vocab-size:32000# The vocabulary size.Now we are ready to create our first dataset. Runlangumo!$langumoThen you can see the below outputs:[*] import file from src/enwiki-20200901-pages-articles1.xml-p1p30303.bz2
[*] parse raw-formatted corpus file with WikipediaParser
[*] merge 1 files into one
[*] shuffle raw corpus file: 100%|██████████████████████████████| 118042/118042 [00:01<00:00, 96965.15it/s]
[00:00:10] Reading files (256 Mo)                   ███████████████████████████████████                 100
[00:00:00] Tokenize words                           ███████████████████████████████████ 418863   /   418863
[00:00:01] Count pairs                              ███████████████████████████████████ 418863   /   418863
[00:00:02] Compute merges                           ███████████████████████████████████ 28942    /    28942
[*] export the processed file to build/vocab.txt
[*] tokenize sentences with WordPiece model: 100%|███████████████| 236084/236084 [00:23<00:00, 9846.67it/s]
[*] split validation corpus - 23609  of 236084 lines
[*] export the processed file to build/corpus.train.txt
[*] export the processed file to build/corpus.eval.txtAfter building the dataset,workspacewould contain the below files:workspace
├── build
│   ├── corpus.eval.txt
│   ├── corpus.train.txt
│   └── vocab.txt
├── src
│   └── enwiki-20200901-pages-articles1.xml-p1p30303.bz2
└── build.ymlUsageusage: langumo [-h] [config]

The unified corpus building environment for Language Models.

positional arguments:
  config      langumo build configuration

optional arguments:
  -h, --help  show this help message and exitDocumentationYou can find thelangumodocumentationon the website.LicenselangumoisApache-2.0 Licensed.