Package: public-datasets
Public datasets pluginThis project implements the public datasets (CIFAR10 / SVHN / MNIST) plugin for the DEEL dataset manager.A deel dataset plugin is an extension of the Dataset class defined in theDEEL dataset manager project.
It allows to access to specific dataset files using theloadmethod and other defined modes.Public datasets (CIFAR10 / SVHN / MNIST) dataset plugin use the default modepathto load.MNIST:train-images-idx3-ubyte.gz,train-labels-idx1-ubyte.gz,t10k-images-idx3-ubyte.gz,t10k-labels-idx1-ubyte.gz,CIFAR10:cifar-10-python.tar.gz,SVHN:housenumbers/train.tar.gz,housenumbers/test.tar.gz,housenumbers/extra.tar.gz,using the http protocol.InstallationThe latest release can be installed from pypi. All needed python packages will also be installed as a dependency.pipinstallpublic-datasetsOtherwize the ssh or HTTPS version should work but you will have to enter your credentials manually:# SSH version (with proper SSH key setup):pipinstallgit+ssh://git@github.com:deel-ai/public_datasets.git# HTTPS version:pipinstallgit+https://github.com/deel-ai/public_datasets.gitNote:CIFAR10 dataset loading name iscifra10,SVHN dataset loading name issvhn,MNIST dataset loading name ismnist.Examples of usageBasic usageTo load one of public datasets (CIFAR10 / SVHN / MNIST), you can simply do:importdeel.datasets# Load the default mode of mnist dataset:mnist_data_path=deel.datasets.load("mnist")# Load the default mode of svhn dataset:svhn_data_path=deel.datasets.load("svhn")# Load the default mode of cifra10 dataset:cifra10_data_path=deel.datasets.load("cifra10")Thedeel.datasets.loadfunction is the basic entry to access the datasets.
By passingwith_info=True, extra information can be retrieved as a python
dictionary. Information are not standardized, so each dataset may provide
different ones:
Themodeargument can be used to load different "version" of the dataset. By default,
only thepathmode is available and will return the path to the local folder
containing the dataset.Command line utilitiesThedeel-datasetspackage comes with some command line utilities that can be accessed using:python -m deel.datasets ARGS...The--helpoption can be used to view the full capabilities of the command line program.
By default, the program uses the configuration at$HOME/.deel/config.yml, but the-cargument can be used to specified a custom configuration file.The following commands are available (not exhaustive):list— List the available datasets. If the configuration specify a remote provider
(e.g., WebDAV), this will list the datasets available remotely. To list the dataset already
downloaded, you can use the--localoption.$python-mdeel.datasetslist
Listingdatasetsathttps://datasets.deel.ai:dataset-a:3.0.1[latest],3.0.0dataset-b:1.0[latest]dataset-c:1.0[latest]$python-mdeel.datasetslist--local
Listingdatasetsat/opt/datasets:dataset-a:3.0.1[latest],3.0.0dataset-c:1.0[latest]download NAME[:VERSION]— Download the specified dataset. If the configuration
does not specify a remote provider, this does nothing except outputing some information.
The:VERSIONcan be omitted, in which case:latestis implied. To force the re-download
of a dataset, the--forceoption can be used.cas de MNIST$python-mdeel.datasetsdownloadmnist
Fetchingmnist...
Matplotlibcreatedatemporaryconfig/cachedirectoryat/tmp/matplotlib-j83d2nc3becausethedefaultpath(/home/<user>/.config/matplotlib)isnotawritabledirectory;itishighlyrecommendedtosettheMPLCONFIGDIRenvironmentvariabletoawritabledirectory,inparticulartospeeduptheimportofMatplotlibandtobettersupportmultiprocessing.
train-images-idx3-ubyte.gz:100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|9.45M/9.45M[00:00<00:00,11.2Mbytes/s]Extractingtrain-images-idx3-ubyte.gz:44.9Mbytes[00:00,248Mbytes/s]train-labels-idx1-ubyte.gz:100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|28.2k/28.2k[00:00<00:00,13.4Mbytes/s]Extractingtrain-labels-idx1-ubyte.gz:58.6kbytes[00:00,132Mbytes/s]t10k-images-idx3-ubyte.gz:100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1.57M/1.57M[00:00<00:00,9.04Mbytes/s]Extractingt10k-images-idx3-ubyte.gz:7.48Mbytes[00:00,246Mbytes/s]t10k-labels-idx1-ubyte.gz:100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|4.44k/4.44k[00:00<00:00,55.2Mbytes/s]Extractingt10k-labels-idx1-ubyte.gz:9.77kbytes[00:00,59.0Mbytes/s]converttrainimages:100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|60000/60000[00:10<00:00,5554.88it/s]converttestimages:100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|10000/10000[00:01<00:00,5526.01it/s]Datasetmnistloadedandstoredat'/home/<user>/.deel/datasets/mnist/1.0.0'.cas de SVHN$python-mdeel.datasetsdownloadsvhn
python-mdeel.datasetsdownloadsvhn
Fetchingsvhn...
Matplotlibcreatedatemporaryconfig/cachedirectoryat/tmp/matplotlib-gl2vzgmibecausethedefaultpath(/home/justin.plakoo/.config/matplotlib)isnotawritabledirectory;itishighlyrecommendedtosettheMPLCONFIGDIRenvironmentvariabletoawritabledirectory,inparticulartospeeduptheimportofMatplotlibandtobettersupportmultiprocessing.
train_32x32.mat:100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|174M/174M[00:47<00:00,3.84Mbytes/s]test_32x32.mat:100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|61.3M/61.3M[00:25<00:00,2.50Mbytes/s]extra_32x32.mat:100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1.24G/1.24G[05:32<00:00,4.00Mbytes/s].....
Datasetsvhnloadedandstoredat'/home/<user>/.deel/datasets/svhn/1.0.0'.