Package: datamaestro
IntroductionFull documentation can be found athttp://datamaestro.rtfd.ioThis projects aims at grouping utilities to deal with the numerous and heterogenous datasets present on the Web. It aims
at beinga reference for available resources, listing datasetsa tool to automatically download and process resources (when freely available)integration with theexperimaestroexperiment manager.(planned) a tool that allows to copy data from one computer to anotherEach datasets is uniquely identified by a qualified name such ascom.lecun.mnist, which is usually the inversed path to the domain name of the website associated with the dataset.The main repository only deals with very generic processing (downloading, basic pre-processing and data types). Plugins can then be registered that provide access to domain specific datasets.List of repositoriesInformation RetrievalNLP and information access related datasetNatural Language Processing (e.g. Sentiment101) and Information access (e.g. TREC) datasetsimage-related datasetImage related datasets (e.g. MNIST)machine learningGeneric machine learning datasetsCommand line interface (CLI)The command line interface allows to interact with the datasets. The commands are listed below, help can be found by typingdatamaestro COMMAND --help:searchsearch dataset by name, tags and/or tasksdownloaddownload files (if accessible on Internet) or ask for download path otherwisepreparedownload dataset files and outputs a JSON containing path and other dataset informationrepositorieslist the available repositoriesorphanslist data directories that do no correspond to any registered dataset (and allows to clean them up)create-datasetcreates a dataset definitionExample (CLI)Retrieve and downloadThe commmand line interface allows to download automatically the different resources. Datamaestro extensions can provide additional processing tools.$datamaestrosearchtag:image[image]com.lecun.mnist

$datamaestropreparecom.lecun.mnist
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gzinto/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-labels-idx1-ubyte
INFO:root:Transformingfile
INFO:root:Createdfile/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-labels-idx1-ubyte
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gzinto/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-images-idx3-ubyte
INFO:root:Transformingfile
INFO:root:Createdfile/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-images-idx3-ubyte
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gzinto/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-labels-idx1-ubyte
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloadinghttp://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz:32.8kB[00:00,92.1kB/s]INFO:root:Transformingfile
INFO:root:Createdfile/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-labels-idx1-ubyte
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gzinto/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-images-idx3-ubyte
INFO:root:Downloadinghttp://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloadinghttp://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz:9.92MB[00:00,10.6MB/s]INFO:root:Transformingfile
INFO:root:Createdfile/home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-images-idx3-ubyte
...JSON...The previous command also returns a JSON on standard output{"train":{"images":{"path":".../data/image/com/lecun/mnist/train_images.idx"},"labels":{"path":".../data/image/com/lecun/mnist/train_labels.idx"}},"test":{"images":{"path":".../data/image/com/lecun/mnist/test_images.idx"},"labels":{"path":".../data/image/com/lecun/mnist/test_labels.idx"}},"id":"com.lecun.mnist"}For those using Python, this is even better since the IDX format is supportedIn[1]:fromdatamaestroimportprepare_datasetIn[2]:ds=prepare_dataset("com.lecun.mnist")In[3]:ds.train.images.data().dtype,ds.train.images.data().shapeOut[3]:(dtype('uint8'),(60000,28,28))Python definition of datasetsEach dataset (or a set of related datasets) is described in Python using a mix of declarative
and imperative statements. This allows to quickly define how to download dataset using the
datamaestro declarative API; the imperative part is used when creating the JSON output,
and is integrated withexperimaestro.Its syntax is described in thedocumentation.For MNIST, this corresponds to.fromdatamaestro_image.dataimportImageClassification,LabelledImages,Base,IDXImagefromdatamaestro.download.singleimportfiledownloaderfromdatamaestro.definitionsimportargument,datatasks,datatags,datasetfromdatamaestro.data.tensorimportIDX@filedownloader("train_images.idx","http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz")@filedownloader("train_labels.idx","http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz")@filedownloader("test_images.idx","http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz")@filedownloader("test_labels.idx","http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz")@dataset(ImageClassification,url="http://yann.lecun.com/exdb/mnist/",)defMNIST(train_images,train_labels,test_images,test_labels):"""The MNIST databaseThe MNIST database of handwritten digits, available from this page, has atraining set of 60,000 examples, and a test set of 10,000 examples. It is asubset of a larger set available from NIST. The digits have beensize-normalized and centered in a fixed-size image."""return{"train":LabelledImages(images=IDXImage(path=train_images),labels=IDX(path=train_labels)),"test":LabelledImages(images=IDXImage(path=test_images),labels=IDX(path=test_labels)),}0.8.0Integration with other repositories: abstracting away the notion of datasetRepository prefixSet sub-datasets IDs automatically0.7.3Updates for new experimaestro (0.8.5)Search types with "type:..."0.6.17Allow remote access through rpyc0.6.9versioncommand