Package: lddl
Language Datasets and Data LoadersLanguage Datasets and Data Loaders (LDDL) is a utility library that minimizes
the friction during dataset retrieval, preprocessing and loading for the
language models inNVIDIA Deep Learning Examples.The current capabilities of LDDL include:Data preprocessing at scale viaDaskandMPI for Python (mpi4py):BERT-PyTorch's andBERT-PaddlePaddle's
dataset for pretraining takes less than 2 mins to preprocess on 32 DGXA100 nodes.BART-PyTorch's
dataset for pretraining takes about 2 hrs to preprocess on 4 DGXA100 nodes.Data loading forPyTorchandPaddlePaddlemulti-node training workloads with minimum overhead.Sequence binning that can reduce end-to-end training latency.InstallationThe steps to install LDDL are listed below:Step 1 [optional but recommended]:Thejemallocmemory allocator is an alternative (to the glibc memory allocator) that might
offer better performance during data preprocessing. On Debian-based Linux distributions (e.g., Ubuntu),
you can install jemalloc via:apt-getinstalllibjemalloc-devStep 2 [required]:LDDL can be installed from the source by runningpip install <target>where<target>is the project root directory of LDDL or
an URL thereof. For examples:pipinstallgit+https://github.com/NVIDIA/lddl.gitorgitclonehttps://github.com/NVIDIA/lddl.git
pipinstall./lddlpipwould also automatically install all LDDL's other Python package
dependencies.LDDL is only tested to work with Python 3!Step 3 [required]:After installingNLTK(either manually or automatically whenpip installLDDL), the model data of
the NLTK's Punkt Sentence Tokenizer needs to be downloaded before the Punkt
Sentence Tokenizer can be used:python-mnltk.downloaderpunktExample DockerfileFor your own tasks, it is highly likely that you would need to use a Docker
container whose image you customize and build. As an example to show how to
install LDDL inside a Docker container, we providea Dockerfilewhich follows the above installation
steps to install LDDL in aNGC Container. You can build
NGC Container images with LDDL installed using this example Dockerfile bybashdocker/build.sh<Dockerfilenamewithoutextension><tagofthebaseimage><outputimagename/URL>NGC Containers arenotone of LDDL's dependencies. You can install LDDL in
your customized Docker image, local virtualenv or conda environments too.PyTorch ContainerTo build the PyTorch image with LDDL installed based on theNGC PyTorch ContainerVersion 21.11and name the output image aslddl:latest:bashdocker/build.shngc_pyt21.11-py3lddl:latestwhere:ngc_pytrefers to usingdocker/ngc_pyt.Dockerfile;21.11-py3means thatnvcr.io/nvidia/pytorch:21.11-py3is used as the base
image;lddl:latestis the name of the built image. After building this image, you
can find its name viadocker image list.PaddlePaddle ContainerTo build the PaddlePaddle image with LDDL installed based on theNGC PaddlePaddle ContainerVersion 22.12and name the output image aslddl:latest:bashdocker/build.shngc_paddle22.12-py3lddl:latestwhere:ngc_paddlerefers to usingdocker/ngc_paddle.Dockerfile;22.12-py3means thatnvcr.io/nvidia/paddlepaddle:22.12-py3is used as the base
image;lddl:latestis the name of the built image. After building this image, you
can find its name viadocker image list.Container UsageYou can launch a container in interactive mode by:bashdocker/interactive.sh<mountspecifications><shellcommand><outputimagename/URL>For example, to launch a container using thelddl:latestimage and mountdatasets/under your home directory to/datasetsinside the container:bashdocker/interactive.sh"-v${HOME}/datasets:/datasets"/bin/bashlddl:latestIntroductionTo clarify, we define the following terms:Offline vs. Online: By "Offline", we mean something to be run as a separate
entity only once with respect to the training jobs; in contrast, "Online" refers
to being run as part of the training jobs.As an analogy, buying kitchen appliances is "Offline" with respect to cooking,
because you only need to buy them once, and the action of buying often happens
when you are not cooking (i.e., when you are going to the stores or shopping
online). In contrast, washing the vegetables is "Online" with respect to
cooking, because it happens every time before you cook the vegetables.Ahead-of-training vs. During-training: By "Ahead-of-training", we mean
something to be run before the training processes start; in contrast,
"During-training" refers to being run during the training processes.In summary, LDDL consists of four components:Stage 1 [Offline]Downloadersthat download the raw text of datasets
from public and online sources.[Offline or Online and Ahead-of-training]Preprocessing:Stage 2Preprocessorsthat preprocesses the raw text into unbalancedParquetshards.Stage 3Load Balancerthat balance the Parquet shards and makes sure
every shard has the same amount of samples.Stage 4 [Online and During-training]Data Loadersthat load the
balanced shards into memory and perform additional preprocessing steps during
training.Depending on the specific usage, a certain step can be performed in different
stages. For example, if you want to experiment with static masking, you can
request the preprocessor to mask each samples; however, if you want to
experiment with dynamic masking, you can request the data loader to mask each
samples.LDDL supports the technique ofsequence binningin order to reduce redundant
computation on the padded tokens:The maximum sequence length is divided into severalbins. For example, if
we want4bins out of a maximum sequence length of512, then:The first bin contains samples whose sequence lengths are between[0, 128];The second bin contains samples whose sequence lengths are between[129, 256];The third bin contains samples whose sequence lengths are between[257, 384];The third bin contains samples whose sequence lengths are between[385, 512];At each training iteration, a bin is randomly selected based on the sequence
distribution of the entire dataset, and all ranks are fed with mini-batches
whose samples all belong to this selected bin.Each mini-batch is only padded to the longest sequence within this
mini-batch.The preprocessor and load balancer can speedup preprocessing large corpora
significantly by scaling to multi-node via Dask and MPI.Meanwhile, the technique of sequence binning can significantly reduce the
end-to-end training latency by reducing redundant computation on the padded
tokens.CoverageDownloaders, preprocessors and the load balancer can be launched via shell
commands:Downloaders:Wikipedia dumps:download_wikipediaBookcorpusdownload_booksCommon Crawldownload_common_crawlOpenWebTextCorpusdownload_open_webtextPreprocessors:BERTPretraining:preprocess_bert_pretrainBARTPretraining:preprocess_bart_pretrainLoad Balancer:balance_dask_outputPlease use the--helpflag to check the exact usage of each command (e.g.,download_wikipedia --help).An implementation of MPI is required to be already installed on your system
for the preprocessors and the load balancer. NGC containers come with a
pre-installed MPI implementation.LDDL currently supports the following data loaders:BERTPyTorchPretraining:lddl.torch.get_bert_pretrain_data_loaderPaddlePaddlePretraining:lddl.paddle.get_bert_pretrain_data_loaderPlease refer to their in-code documentation for more details.Quick Start GuideBERTWe provide two working example scripts to demonstrate how to use LDDL end-to-end
(i.e., from downloading the datasets to loading input batches during training)
for a (mock) BERT Phase 2 pretraining task:Running on a local machine:local_example.sh.
You can run this script bybash examples/local_example.sh.Running on a Slurm cluster and scale to multi-nodes:slurm_example.sub. Before running this script,
you need to download and move the datasets to the right location in the NFS of
your Slurm cluster. You might also need to customize this script to match the
specific settings of your Slurm cluster. You can run this script and submit
jobs to Slurm bysbatch -N<number of nodes> examples/slurm_example.sub(e.g.,sbatch -N2 examples/slurm_example.subif you want to
run on 2 nodes).We assume that these two example scripts could be run without interruption so
that they would work out-of-the-box. You can also comment out the commands in
these two scripts to run each step individually. Important steps in the above
working example scripts are summarized and highlighted below:Offline DownloaderThe Wikipedia corpus can be downloaded via:download_wikipedia--outdir<Wikipediaoutputpath>where<Wikipedia output path>is where you want to store the raw text of the
Wikipedia corpus. For example,download_wikipedia--outdirdata/wikipediawould download the raw text of the Wikipedia corpus todata/wikipedia.Offline, or Online and Ahead-of-training PreprocessingPreprocessorThe dataset for BERT pretraining can be preprocessed via:mpirun\-np$(nproc)\--oversubscribe\--allow-run-as-root\-xLD_PRELOAD=<pathtolibjemalloc.so>\preprocess_bert_pretrain\--schedulempi\--target-seq-length<128forPhase1;512forPhase2>\--wikipedia<Wikipediaoutputpath>/source\--sink<BERTpretraininginputpath>\--vocab-file<pathtothevocabfile>\--num-blocks<numberofinputshards>\--bin-size<binsize>where:<BERT pretraining input path>is where you want to store the preprocessed
but unbalanced Parquet shards generated by this BERT pretraining preprocessor.<path to the vocab file>is the path to thevocab filethat is used by the WordPiece tokenizer.<number of input shards>is the total number of input shards; this number
needs to be a positive integer multiple of<world size> * <max(DataLoader's num_workers, 1)>.if you want to enable sequence binning, you can set the--bin-sizeflag. Thei-th bin contains sequences that have from(i - 1) * <bin size> + 1toi * <bin size>tokens.<path to libjemalloc.so>will depend on your Linux distribution. For
Debian-based distributions, it can be found at/usr/lib/x86_64-linux-gnu/libjemalloc.so.If you want to use the memory allocator from glibc instead of jemalloc, you
can omit the-x LD_PRELOAD=<path to libjemalloc.so>flag tompirun. Either
jemalloc or glibc could be more suitable to your specific system, so we
recommend trying both and select the one that yields the best performance.By default, masking is deferred to the online and during-training data loading
and performed dynamically. During dynamic masking, each sample can be masked
differently among epochs. If you want to enable static masking that takes place
during the preprocessor stage, you need to add the--maskingflag.For example,mpirun\--oversubscribe\--allow-run-as-root\-np64\-xLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so\preprocess_bert_pretrain\--schedulempi\--vocab-filedata/vocab/bert-en-uncased.txt\--wikipediadata/wikipedia/source/\--sinkdata/bert/pretrain/phase2/bin_size_64/\--target-seq-length512\--num-blocks4096\--bin-size64\--maskingwould use a total of64processes to run the preprocessor for BERT Phase 2
pretraining (whose maximum sequence length is512). In this case, the shared
object of jemalloc is located at/usr/lib/x86_64-linux-gnu/libjemalloc.so; the vocab file
is located atdata/vocab/bert-en-uncased.txt; the Wikipedia corpus is
downloaded atdata/wikipedia/; the preprocessor would store the unbalanced
(roughly4096) Parquet shards atdata/bert/pretrain/phase2/bin_size_64/;
the sequence binning is enabled with the bin size of64; and the static
masking is enabled as well.Load BalancerWe can balance the number of samples among the preprocessed but unbalanced
Parquet shards via:mpirun-np$(nproc)--oversubscribe--allow-run-as-root\balance_dask_output\--indir<BERTpretraininginputpath>\--num-shards<numberofinputshards>After the load balancer finishes, all shards will have exactly the same number
of samples, or some shards are different by only 1 sample if the total number of
shards does not divide the total number of samples. If you don't specify a path
to the--outdirflag, the Parquet shards in<BERT pretraining input path>will be modified in-place. For example,mpirun\--oversubscribe\--allow-run-as-root\-np64\balance_dask_output\--indirdata/bert/pretrain/phase2/bin_size_64/\--num-shards4096would use64processes to run the load balancer which balances the Parquet
shards (located atdata/bert/pretrain/phase2/bin_size_64/) into exactly4096shards. Among these shards, the number of samples could differ by at most 1.The abovempiruncommands show how the preprocessor and load balancer could
be run on a single node. The flags passed intompirunoften need to be
adjusted based on the configuration of your compute cluster.We also provide anexampleto demonstrate how to
run the preprocessor and load balancer on SLURM clusters that supportMPI,PyxisandEnroot.Online and During-training Data LoadingPyTorchWe can get the LDDL dataloader for BERT pretraining vialddl.torch.get_bert_pretrain_data_loader(please refer to thein-code documentationof this function). Afterwards,
we can use it like a normal PyTorch DataLoader instance. For example,importargparseimportloggingimportosimportlddl.torchparser=argparse.ArgumentParser()parser.add_argument('--local_rank',type=int,default=os.getenv('LOCAL_RANK',0),help='local_rank is set by torch.distributed.launch or SLURM',)args=parser.parse_args()# Contains the balanced Parquet shards generated by the load balancer.input_dir='data/bert/pretrain/phase2/bin_size_64/'# Path to the vocab file.vocab_file='data/vocab/bert-en-uncased.txt'# Number of samples in a single mini-batch per rank.batch_size=64# Number of DataLoader worker processes per rank.num_workers=4# Epoch number to start with.start_epoch=0# Total number of epochs to train. One epoch refers to going through the entire# dataset once.epochs=2train_dataloader=lddl.torch.get_bert_pretrain_data_loader(input_dir,local_rank=args.local_rank,vocab_file=vocab_file,data_loader_kwargs={'batch_size':batch_size,'num_workers':num_workers,'pin_memory':True,},log_level=logging.WARNING,start_epoch=start_epoch,)...forepochinrange(start_epoch,start_epoch+epochs):fori,batchinenumerate(train_dataloader):prediction_scores,seq_relationship_score=model(input_ids=batch['input_ids'].to(device),token_type_ids=batch['token_type_ids'].to(device),attention_mask=batch['attention_mask'].to(device),)loss=criterion(prediction_scores,seq_relationship_score,batch['labels'].to(device),batch['next_sentence_labels'].to(device),)...We provide a(mock) training scriptthat shows how
the LDDL dataloader should be used. For example, if the balanced Parquet shards
are located atdata/bert/pretrain/phase2/bin_size_64/and the vocab file is
located atdata/vocab/bert-en-uncased.txt, you can run this (mock) training
script with a world size of2on a single machine via:python-mtorch.distributed.launch--nproc_per_node=2\benchmarks/torch_train.py\--pathdata/bert/pretrain/phase2/bin_size_64/\--vocab-filedata/vocab/bert-en-uncased.txtOnce the (mock) training processes are up and running, and the first rank starts
to print output, these processes simply emulate the training loop which could
take some time to go through one epoch. You can kill these processes at any
time.PaddlePaddleWe can get the LDDL dataloader for BERT pretraining vialddl.paddle.get_bert_pretrain_data_loader(please refer to thein-code documentationof this function). Afterwards,
we can use it like a normal PaddlePaddle DataLoader instance. For example,importloggingimportosimportlddl.paddle# Contains the balanced Parquet shards generated by the load balancer.input_dir='data/bert/pretrain/phase2/bin_size_64/'# Path to the vocab file.vocab_file='vocab/bert-large-uncased-vocab.txt'# Number of samples in a single mini-batch per rank.batch_size=64# Number of DataLoader worker processes per rank.num_workers=4# Epoch number to start with.start_epoch=0# Total number of epochs to train. One epoch refers to going through the entire# dataset once.epochs=2train_dataloader=lddl.paddle.get_bert_pretrain_data_loader(input_dir,vocab_file=vocab_file,data_loader_kwargs={'batch_size':batch_size,'num_workers':num_workers,},log_level=logging.WARNING,start_epoch=start_epoch,)...forepochinrange(start_epoch,start_epoch+epochs):fori,batchinenumerate(train_dataloader):prediction_scores,seq_relationship_score=model(input_ids=batch['input_ids'],token_type_ids=batch['token_type_ids'],attention_mask=batch['attention_mask'],)loss=criterion(prediction_scores,seq_relationship_score,batch['masked_lm_labels'],batch['next_sentence_labels'],)...We provide a(mock) training scriptthat shows how
the LDDL dataloader should be used. For example, if the balanced Parquet shards
are located atdata/bert/pretrain/phase2/bin_size_64/and the vocab file is
located atvocab/bert-large-uncased-vocab.txt, you can run this (mock) training
script with a world size of2on a single machine via:python-mpaddle.distributed.launch--gpus=0,1\benchmarks/paddle_train.py\--pathdata/bert/pretrain/phase2/bin_size_64/\--vocab-filevocab/bert-large-uncased-vocab.txtOnce the (mock) training processes are up and running, and the first rank starts
to print output, these processes simply emulate the training loop which could
take some time to go through one epoch. You can kill these processes at any
time.ContributionWe welcome any form of contribution! The simplest contribution would be to try
LDDL on your own NLP tasks where data preprocessing and loading is a headache
for you. If you find rough edges for your specific use case, please file a
GitHub issue.