Package: infinibatch
InfiniBatchInfinibatch is a library of checkpointable iterators for randomized data loading of massive data sets in deep neural network training.Featuressupport for corpora much larger than fit into RAMhierarchical block+sentence-level randomization over the whole corpus, different randomization in each epochonly load the data that is neededvery fast start-up time (does not need to read full corpus)only requires the most basic of data preparation (e.g. no indexing)for multi-GPU, only load what the respective GPU needs100% accurate check-pointing, restore from checkpoint should not read all data up to the checkpointsupport automatic bucketed batching with dynamic batch sizespre-fetching threadcomposable, as to support for complex batching, e.g. negative samples from multiple documentsGetting StartedInfinibatch requires Python 3.6 or higher and has no dependencies.
There is presently no pip package.To install it, clone this repository and install it locally.gitclonehttps://github.com/microsoft/infinibatchcdinfinibatch
pipinstall-e.DocumentationThe documentation can be found here:https://microsoft.github.io/infinibatch/TutorialThis little tutorial walks you through the steps of preparing your data and consuming them from Python code as batches.Infinibatch Basics: Iterators and CheckpointingInfinibatch providesPython iteratorsto read your data.
An iterator represents a stream of data that can be retrieved item by item, e.g. via aforloop or repeatedly callingnext()on it.Infinibatch is agnostic to the data type of the items, which is determined by a user-supplied file-read function.
In NLP applications, items would typically be tuples of text. In other applications,
they can be images or an audio file with a textual annotation.Infinibatch makes it easy to read your data in randomized order, and supports checkpointing, which allows you to restart training exactly where you left off.Randomization is doneon the fly, which means that it is not necessary to read the entire data set into memory
to be shuffled. Infinibatch implements a hierarchical shuffling algorithm
that only holds a subset of the data in RAM at any point in time.Infinibatch iterators arecheckpointable.
Checkpointing lets you retrieve the current position (the "checkpoint") in the data stream at any time, so that
later, you can "rewind" to that same position.
The sad reality is that long-running trainings occasionally crash.
To be able to continue a crashed training as if it had not crashed,
save your Infinibatch iterator's checkpoint to disk whenever you save an intermediate model during training.
To restart a crashed training, reset the iterator to the saved checkpoint.
The data reader will now yield the exact same data-item sequence it would have yielded without the crash.Data PreparationInfinibatch has one requirement on your data organization:
To use your data with Infinibatch, it must be split into a large number of small chunks.
A chunk is the smallest unit of data that is loaded from disk into RAM. Infinibatch holds a random subset of chunks in memory
that it randomly draws samples from.Below we want to show how such a split can be created. An easy way to split your data into chunks is with the Linuxsplitcommand.In this tutorial, our "corpus" consists of 6 lines of text, where each line is one data item.
To create that corpus, please run this command in a bash shell. It creates a 6-line text file namedcorpus.txt:echo\\'Lorem ipsum dolor sit amet,consectetur adipiscing elit,sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.The quick brown fox jumps over the lazy dog.'\\>corpus.txtNow let us split it into 3 chunks of 2 lines each. Each chunk is stored as a zipped text file.
We will create them inside a new subdirectory calledcorpus_chunks:mkdircorpus_chunks
split--lines2--numeric-suffixes\\--filter'gzip > corpus_chunks/$FILE.txt.gz'\\corpus.txtcorpus.This will have created three files:corpus_chunks/corpus.00.txt.gz,corpus_chunks/corpus.01.txt.gz, andcorpus_chunks/corpus.02.txt.gz.
To verify whether the data has been split as expected, you can use this command:zcatcorpus_chunks/corpus.*.txt.gzHint: For large corpora, we recommend replacinggzipbypigz(apt-get install pigz), which runs notably faster via multi-threading.Reading Items in Random Order With InfinibatchWe will first show the easiest way to read data with Infinibatch, using the helper functionchunked_dataset_iterator``().
This function will create an Infinibatch iterator that yields the content of your data in random order.
Please the following program:importgzip,globfrominfinibatchimportdatasetsasdsds=ds.chunked_dataset_iterator(chunk_refs=glob.glob('corpus_chunks/corpus.*.txt.gz'),read_chunk_fn=lambdapath:iter(gzip.decompress(open(path,"rb")\\.read()).decode(encoding='utf-8')\\.splitlines()),buffer_size=6,seed=1)foriinrange(10):print(next(ds))You should get output that contains the 6 example lines in randomized order:Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
The quick brown fox jumps over the lazy dog.
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
consectetur adipiscing elit,
Lorem ipsum dolor sit amet,
The quick brown fox jumps over the lazy dog.
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Note: Thebuffer_sizeparameter determines how many sentences are read into memory at any given time,
to draw randomized items from. In real settings with corpora of hundreds of millions of text lines,
thebuffer_sizeparameter should be set in the millions.
RAM usage and startup time will be proportional to the buffer size
(but much lower than having to load the entire corpus into RAM).Reading Items of Different Lengths in BatchesFor deep learning, we want to group multiple items into batches.
For NLP tasks, items are often lines of text of varying length.
Infinibatch implements an algorithm that randomizes the input sequence and groups it into
batches of approximately the same length (akabucketing).Infinibatch'sBucketedReadaheadBatchIteratorperforms this task.
It implements an algorithm modeled after theMarian toolkitthat preloads a large number of randomized items (typically millions; in this example: 6),
sorts them and groups them into batches of similar length, and then yields
them, in turn, in randomized order.Here is an example. Note that theBucketedReadaheadBatchIteratoraccepts
the previous randomized sentence sequence iterator (ds) as the source of items to randomize over.
This is an example how one forms pipelines of iterators with Infinibatch
(a concept familiar from Python's ownitertools).
Once an iterator is passed to another as its source, consider it owned by that other iterator,
it must no longer be accessed by the calling code.importgzip,globfrominfinibatchimportdatasetsasdsfrominfinibatchimportiteratorsasitds=ds.chunked_dataset_iterator(chunk_refs=glob.glob('corpus_chunks/corpus.*.txt.gz'),read_chunk_fn=lambdapath:iter(gzip.decompress(open(path,"rb")\\.read()).decode(encoding='utf-8')\\.splitlines()),buffer_size=6,seed=1)bs=it.BucketedReadaheadBatchIterator(source_iterator=ds,# note: this is the iterator from aboveread_ahead=6,key=lambdaline:len(line),batch_size=2,seed=1)foriinrange(25):print(next(bs))This code should output something like this:['sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.','The quick brown fox jumps over the lazy dog.']['consectetur adipiscing elit,','Lorem ipsum dolor sit amet,']['Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.','Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.']followed by different permutations of the same tuples.
As you can see, the sentences are in random order and grouped in batches of 2 of approximately the same length.
You may notice that there is no variation in how the items get grouped into batches--that
is an artifact of this example, and generally not the case in real use when the data size is much larger
than the batch size.In NLP, sentence length often varies considerably. As a result, using batches of a fixed number of lines,
as in the example above, will waste GPU RAM and cores.
This is because the number of lines is limited by the longest possible sequence; batches of shorter lines
would leave GPU cycles on the table.
Ideally, one would use batches that have as many lines as fit into GPU RAM,
given the number of tokens of the longest line in the batch.
To support variable batch sizes, Infinibatch allows to pass a function as thebatch_sizeparameter.
That function will be given the longest item of a batch and should estimate how many items of at most this length can fit.In our example, we assume that batches can hold at most 150 tokens.
Please change the above code as follows:batch_size=lambdalongest_line:150//len(longest_line),The output looks like this:['consectetur adipiscing elit,', 'Lorem ipsum dolor sit amet,']
['Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.']
['sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.',
 'The quick brown fox jumps over the lazy dog.']
['Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.']That shorter sentences got grouped, while longer did not because they would exceed the total of 150 characters.Reading Batches Into Numpy ArraysLastly, we will need to feed batches into our favorite deep-learning tool.
We will show how to convert the batches of text lines into paddednumpyarrays.In a typical NLP application, text items would be tokenized, and then each token
would be represented by an index into a unit vocabulary.
For simplicity, in this example each character is its own token,
and each token's numeric unit index is just its ASCII code.
These sequences are then padded to equal length with -1, and converted into anumpyarray.Please rerun the previous example, but first insert the following code before the finalforloop.
This example uses an InfinibatchMapIterator, which applies a user-supplied function or
lambda to each item:importnumpyasnpdefcollate(lines_batch):# tokenize all lines in the batch and map to unit idsids_batch=[[ord(c)forcinline]forlineinlines_batch]# create a padded numpy array as wide as the longest line,# where shorter sequences are padded with -1width=max(len(ids)foridsinids_batch)returnnp.array([ids+[-1]*(width-len(ids))foridsinids_batch])bs=it.MapIterator(source_iterator=bs,transform=collate)This will output batches like this. Note that in batches with multiple sentences,
some entries are padded with-1.[[99111110115101991161011161171143297100105112105115991051101033210110810511644][76111114101109321051121151171093210011110811111432115105116329710910111644-1]][[8511632101110105109329710032109105110105109321181011101059710944321131171051153211011111511611411710032101120101114991051169711610511111032117108108971099911132108979811111410511532110105115105321171163297108105113117105112321011203210197329911110910911110011132991111101151011131179711646]][[115101100321001113210110511711510911110032116101109112111114321051109910510010510011711011632117116321089798111114101321011163210011110811111410132109971031109732971081051131179746][84104101321131171059910732981141111191103210211112032106117109112115321111181011143211610410132108971221213210011110346-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1]][[681171051153297117116101321051141171141013210011110811111432105110321141011121141011041011101001011141051163210511032118111108117112116971161013211810110810511632101115115101329910510810811710932100111108111114101321011173210211710310597116321101171081089732112971141059711611711446]]Where To Go From HereThe above tutorial showed you the use of the most common iterator type, as created by the
convenience functionchunked_dataset_iterator().Not all real-life scenarios are covered by this function. For example, multi-task learning
scenarios require more complex combinations of data. To create those, you will need
to compose the necessary data reader from the underlying building blocks.
This is described at the documentation of the moduleiterators.DocumentationTo view the documentation, please clone the repository and go to docs/infinibatch/index.htmlWhen working on the documentation, install pdoc:pip install pdoc3You can then start a local http server that dynamically updates the documentation:pdoc --template-dir docs --http : infinibatchWe currently haven't set up the CI to automatically generate the documentation.
Before you merge anything into master, please delete the existing documentation in docs/infinibatch and runpdoc -o docs --template-dir docs --html infinibatchTestingTo run unit tests, run the following command.python -m unittest discover -s testIf you would like the unit tests to stop after the first failed test, use:python -m unittest discover -s test --failfastTo type-check withmypy(if installed):mypy infinibatchContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.