Package: yogadl
Yoga Data Layer: TheFlexibleData LayerA better approach to data loading for Deep Learning.  API-transparent caching to disk, GCS, or S3.Whyyogadl?AtDetermined AI, we help many of our customers perform high-performance data
loading for deep learning models.  We believe every data loader should have two layers: therandom-access layerand thesequential layer.Therandom-access layeris critical for good training infrastructure.  Direct random access to
any record enables:Shuffling (potentially every epoch)Pausing/continuing training mid-epochSharding the dataset efficiently for distributed trainingThesequential layerstarts as soon as you decide the order in which you will access the records in
the dataset.  Often the transition is implicit, in which case it starts as soon as you are done
modifying the order of access (i.e. via shuffling, sharding, or splitting).  This layer is vital to
performance optimizations because it enables:Prefetching data loading to hide latency costsParallelizing data loading to hide compute costsHere is a simple code snippet to illustrate what the transition from random-access layer to
sequential layer looks like:# Start of random-access layer.indices=list(range(100))indices=indices[skip:]indices=np.random.shuffle(indices)# Start of sequential layer.defrecord_gen():foriinindices:yieldread_file_at_index(i)record_ds=tf.data.Dataset.from_generator(record_gen,...)final_ds=record_ds.prefetch(...)Notice that in the above example, thetf.dataAPI is used, but only in the sequential layer.
This is becausetf.datahas no concept of the random access layer.  As a result:tf.data.Dataset.shuffle()can only approximate a shuffle.  Calling.shuffle(N)will readNrecords into a buffer and choose samples randomly fromthoseNrecords, while a true
shuffle chooses samples randomly from theentire dataset.  This shortcoming forces you
to choose between memory footprint and the quality of your shuffle.  The only true
shuffle with tf.data.Dataset.shuffle() is to read the entire dataset into memory.tf.data.Dataset.skip(N)is as inefficient as possible.  Each of theNskipped records will
still be read from disk and processed normally, according to all of the operations preceeding
the.skip()call, making.skip()prohibitively expensive for most use cases.Pausing and continuing training is only possible by saving the state of atf.data.Iterator.
However, saving atf.data.Iteratordoes not work with all datasets.  In particular, it does
not work with datasets created usingfrom_generator(), which is the easiest way to create atf.data.Dataset.We have seen countless instances wheretf.data.Datasetshortcomings have made life harder for
deep learning practitioners, so we set out to build something better.  We set out to build a new
data layer which could augment an existingtf.data.Datasetdata loader with the properties should
come standard with every data loader.At the same time, we wanted this new data layer to relieve another key pain point: high-performance
dataset caching and dataset versioning.What isyogadl?We designedyogadlto be two things: a standalone caching layer to imbue existing data loaders
with the properties that come from a random-access layer, and a better interface for defining data
loaders in general.A standalone caching toolSincetf.data.Dataset-based datasets have no random-access layer,yogadlcaches them to disk in
a random-access-friendly way.  The storage mechanism is, in fact, nearly identical to howTensorPack caches datasets to disk,
only with some additional abstractions to allow dataset versioning, cloud storage, and all of the
wonderful features that a data loader with a random-access layer ought to have.What does all this do for you?  A few things:Better training: Ayogadl-cachedtf.data.Datasetwill have better shuffling than a
nativetf.data.Dataset.  Additionally, pausing and continuing training mid-epoch will be
simple and robust, and efficient sharding for distributed training comes standard.Faster data loading: Slow data loader?  Don't waste your time optimizing it.yogadlwill
save it in a high-performance cache the first time it is used, and all future uses will be
fast and efficient.API-transparent: Not all operations in the data loader are cacheable.  Data augmentation
must be done at run time.yogadlallows you to keep your existing data augmentation code.A better interfaceAt the core ofyogadlis theDataRefinterface, which creates an explicit boundary between the
random-access layer and the sequential layer.We are not the first people to think of this: PyTorch separates theDataSet(the random-access
layer) from theSampler(which defines the sequential layer).  Keras has aSequenceobject
which defines the random-access layer, leaving the order of access (the sequential layer) to be
decided by the arguments tomodel.fit().  BothDataSetandSequenceare already 100%
compatible withyogadl'sDataRefinterface (althoughyogadldoes not yet include those
adapters).And yet, the world is still full of data loaders which are lacking.  At Determined AI, we are
dedicated to advancing the state of the art for training Deep Learning models, and we believe that
a better interface for data loading is a critical piece of that goal.  Any data loader which
implements theDataRefinterface is capable of proper shuffling, pausing and continuing training
mid-epoch, and efficient multi-machine distributed training.What isyogadlnot?yogadlis not a data manipulation API.Thisworldhasmorethanenoughofthose.
Instead,yogadlseeks to be API-transparent so that you can continue to use your existing data
loading code, but with all the benefits of a high-performance, random-access cache.  If you have
data augmentation steps which cannot be cached, that code should continue to work without any
modifications.yogadldoes not (at this time) work with any data frameworks other thantf.data.Dataset.First-class support for (tf.)KerasSequenceobjects, PyTorchDataSetobjects, and TensorPackDataFlowobjects is on the near-term roadmap.yogadloffers basic dataset versioning, but it is not (at this time) a full-blown version control
for datasets.  Offering something like version control for datasets is on the roadmap as well.Installingyogadlyogadlcan be installed viapip install yogadl.Further InformationPlease refer to the following links for more information:YogaDL official documentationYogaDL examplesYogaDL API Reference