Package: datadings
datadings is a collection of tools to prepare datasets for machine
learning, based on two simple principlesDatasets are collections of individual data samples.Each sample is a dictionary with descriptive keys.For supervised training with images samples are dictionaries like this:{"key": unique_key, "image": imagedata, "label": label}msgpackis used as an efficient storage
format for most supported datasets.Check out thedocumentationfor more details.Supported datasetsDatasetShort DescriptionADE20kScene Parsing, SegmentationANP460own Eye-Tracking dataset (Jalpa)CAMVIDMotion-based SegmentationCAT2000MIT SaliencyCIFAR32x32 color image classification with 10/100 classesCityscapesSegmentation, Semantic understanding of urban street
scenesCoutrot1Eye-Tracking, SaliencyFIGRIMFixationEye-Tracking, SaliencyILSVRC2012Imagenet Large Scale Visual Recognition ChallengeImageNet21kA superset of ILSVRC2012 with 11 M images for 10450 classesInriaBuildingsInria Areal Image Labeling Dataset (Buildings),
Segmentation, Remote SensingMIT1003Eye-Tracking, Saliency, Learning to predict where
humans lookMIT300Eye-Tracking, SaliencyPlaces2017MIT Places, Scene RecognitionPlaces365MIT Places365, Scene RecognitionRIT18High-Res Multispectral Semantic Segmentation,
Remote SensingSALICON2015Saliency in Context, Eye-TrackingSALICON2017Saliency in Context, Eye-TrackingVOC2012Pascal Visual Object Classes ChallengeVaihingenRemote Sensing, Semantic Object Classification,
SegmentationYFCC100mYahoo Flickr Creative Commons 100 M picsCommand line toolsdatadings-writecreates new dataset files.datadings-catprints the (abbreviated) contents of a dataset file.datadings-shuffleshuffles an existing dataset file.datadings-mergemerges two or more dataset files.datadings-splitsplits a dataset file into two or more subsets.datadings-benchruns some basic read performance benchmarks.Basic usageEach dataset defines modules to read and write in thedatadings.setspackage.
For most datasets the reading module only contains additional
metadata like class labels and distributions.Let’s consider theMIT1003dataset as an example.MIT1003_writeis an executable that creates dataset files.
It can be called directly or throughdatadings-write.
Three files will be written:MIT1003.msgpackcontains sample dataMIT1003.msgpack.indexcontains index for random accessMIT1003.msgpack.md5contains MD5 hashes of both filesReading all samples sequentially,
using aMsgpackReaderas a context manager:with MsgpackReader('MIT1003.msgpack') as reader:
    for sample in reader:
        [do dataset things]This standard iterator returns dictionaries.
Use therawiter()method to get samples as messagepack encoded
bytes instead.Reading specific samples:reader.seek_key('i14020903.jpeg')
print(reader.next()['key'])
reader.seek_index(100)
print(reader.next()['key'])Reading samples as raw bytes:raw = reader.rawnext()
for raw in reader.rawiter():
    print(type(raw), len(raw))Number of samples:print(len(reader))You can also change the order and selection of iterated samples
with augments.
For example, to randomize the order of samples, wrap the reader
in aShuffler:from datadings.reader import Shuffler
with Shuffler(MsgpackReader('MIT1003.msgpack')) as reader:
    for sample in reader:
        # do dataset things, but in random order!A common use case is to iterate over the whole dataset multiple times.
This can be done with theCycler:from datadings.reader import Cycler
with Cycler(MsgpackReader('MIT1003.msgpack')) as reader:
    for sample in reader:
        # do dataset things, but FOREVER!