Package: mads-datasets
MADS Datasets LibraryThis library provides the functionality to download, process, and stream several datasets.InstallationThis library has been published on PyPi and can be installed with pip, conda, pdm or poetry.# Install with pippipinstallmads_datasets# Install with poetrypoetryaddmads_datasets# install with pdmpdmaddmads_datasetsData TypesCurrently, it supports the following datasets:SUNSPOTS Time-Series data, 3000 monthly sunspot observations from 1749IMDB Text data, 50k movie reviews with positive and negative sentiment labelsFLOWERS Image data, about 3000 large and complex images of 5 flowersFASHION MNIST Image data, 60k images sized 28x28 pixelsGESTURES Time-Series data with x, y and z accelerometer data for 20 gestures.IRIS dataset, 150 observations of 4 features of 3 iris flower speciesPENGUINS dataset, an alternative to Iris with 344 penguins on multiple islands.FAVORITA dataset, 125 million sales records of 50k products in 54 stores.An additional type is SECURE, which is used for datasets that are not publicly available. See examples below.UsageAfter installation, import the necessary components:frommads_datasetsimportDatasetFactoryProvider,DatasetTypeYou can create a specific dataset factory using theDatasetFactoryProvider.For instance, to create a factory for the Fashion MNIST dataset:fashion_factory=DatasetFactoryProvider.create_factory(DatasetType.FASHION)With the factory, you can download the data, create datasets and provide the datasets wrapped in datastreamers in one command:streamers=mnistfactory.create_datastreamer(batchsize=32)train=streamers["train"]X,y=next(train.stream())This will do multiple things:it downloads the data to the default location~/.cache/mads_datasetsif not already presentit checks the hash of the fileit transforms the file into a dataset that implements the DatasetProtocol:classDatasetProtocol(Protocol):def__len__(self)->int:...def__getitem__(self,idx:int)->Any:...it creates a streamer.The train.stream() command wil return a generator that yields batches of data.You could also create a dataset directly:dataset=fashion_factory.create_dataset()Or just download the data:fashion_factory.download_data()ExamplesDefault cache locationfashion_factory.datadirThis shows you the default location of all datasets. Storing the datasets in a central place makes
it much easier to manage your storage over multiple projects where you might want to reuse the
same dataset multiple times.It also makes it much easier to clean you storage, as you can just delete the entire directory.In general, it is bad practice to store the raw data in your git repo. Using a central location avoids this (as would adding yourdatafolder to.gitignore).
Instead of storing data in git, provide the user
with a scriptthat downloads the raw datapreprocesses the raw data into something that fits your needsIf you want to change the default location of the cache, give the factory adatadirargument like this:frompathlibimportPathfashion_factory=DatasetFactoryProvider.create_factory(dataset_type=DatasetType.FASHION,datadir=Path("~/path/to/alternative/folder"))Now, instead of using~/.cache/mads_datasets, the data will be stored in~/path/to/alternative/folderwhich can be anywhere you have access to.using Secure datasetsfrom mads_datasets.settings import SecureDatasetSettings
from mads_datasets import DatasetFactoryProvider, DatasetType

garbagesettings = SecureDatasetSettings(
    dataset_url="https://gitlab.com/api/v4/projects/12345/repository/files/filename.extension/raw?lfs=true",
    filename="garbage.parq",
    name="garbage",
    keyaccount="gitlab-MADS-PAT",
    keyname="gitlab-MADS-PAT",
    digest="b5ee4ab8723e0d97e0eefa12e347d04e",
    unzip=False,
)
garbagedata = DatasetFactoryProvider.create_factory(settings=garbagesettings, dataset_type=DatasetType.SECURE)
garbagedata.download_data()For secure datasets, you will have to provide aSecureDatasetSettingsobject. Currently, this is implemented for the gitlab API.You will have to provide an url where you change 12345 with the project ID (settings>general). You should also create a PAT (settings> access tokens).Changefilename.extensionwith your filename (eg.garbage.parq).
The first time you run this, you will have to provide your PAT. It will be stored in your keyring asgitlab-MADS-PAT, or whatever you set askeyaccountandkeyname.The digest can be left out; the first time you run this, you will get the digest and you can add it to make sure the file is not corrupted.If your file needs unzipping, setunzip=True.