Package: datasetops
Dataset Ops: Fluent dataset operations, compatible with your favorite librariesDataset Ops provides afluent interfaceforloading, filtering, transforming, splitting,andcombiningdatasets.
Designed specifically with data science and machine learning applications in mind, it integrates seamlessly withTensorflowandPyTorch.Appetizerimportdatasetopsasdo# prepare your datatrain,val,test=(do.from_folder_class_data('path/to/data/folder').named("data","label").image_resize((240,240)).one_hot("label").shuffle(seed=42).split([0.6,0.2,0.2]))# use with your favorite frameworktrain_tf=train.to_tensorflow()train_pt=train.to_pytorch()# or do your own thingforimg,labelintrain:...InstallationBaniry installers available at thePython package indexpipinstalldatasetopsWhy?Collecting and preprocessing datasets is tiresome and often takes upwards of 50% of the effort spent in the data science and machine learning lifecycle.
WhileTensorflowandPyTorchhave some useful datasets utilities available, they are designed specifically with the respective frameworks in mind.
Unsuprisingly, this makes it hard to switch between them, and training-ready dataset definitions are bound to one or the other.
Moreover, they do not aid you in standard scenarios where you want to:Sample your dataset non-random ways (e.g with a fixed number of samples per class)Center, standardize, normalise you dataCombine multiple datasets, e.g. for parallel input to a multi-stream networkCreate non-standard data splitsDataset Opsaims to make these processing steps easier, faster, and more intuitive to perform, while retaining full compatibility to and from the leading libraries. This also means you can grab a dataset fromtorchvision datasetsand use it directly with tensorflow:importdoimporttorchvisiontorch_usps=torchvision.datasets.USPS('../dataset/path',download=True)tensorflow_usps=do.from_pytorch(torch_usps).to_tensorflow()Development StatusThe library is still under heavy development and the API may be subject to change.What follows here is a list of implemented and planned features.LoadersLoader(utility class used to define a dataset)from_pytorch(load from atorch.utils.data.Dataset)from_tensorflow(load from atf.data.Dataset)from_folder_data(load flat folder with data)from_folder_class_data(load nested folder with a folder for each class)from_folder_dataset_class_data(load nested folder with multiple datasets, each with a nested class folder structure )from_mat(load contents of a .mat file as a single dataaset)from_mat_single_mult_data(load contents of a .mat file as multiple dataasets)load(load data from a path, automatically inferring type and structure)Convertersto_tensorflow(convert Dataset into tensorflow.data.Dataset)to_pytorch(convert Dataset into torchvision.Dataset)Dataset informationshape(get shape of a dataset item)counts(compute the counts of each unique item in the dataset by key)unique(get a list of unique items in the dataset by key)named(supply names for the item elements)names(get a list of names for the elements in an item)stats(provide an overview of the dataset statistics)origin(provide an description of how the dataset was made)Sampling and splittingshuffle(shuffle the items   in a dataset randomly)sample(sample data at random a dataset)filter(filter the dataset using a predicate)split(split a dataset randomly based on fractions)split_filter(split a dataset into two based on a predicate)allow_unique(handy predicate used for balanced classwise filtering/sampling)take(take the first items in dataset)repeat(repeat the items in a dataset, either itemwise or as a whole)Item manipulationreorder(reorder the elements of the dataset items (e.g. flip label and data order))transform(transform function which takes other functions and applies them to the dataset items.)categorical(transforms an element into a categorical integer encoded label)one_hot(transforms an element into a one-hot encoded label)numpy(transforms an element into a numpy.ndarray)reshape(reshapes numpy.ndarray elements)image(transforms a numpy array or path string into a PIL.Image.Image)image_resize(resizes PIL.Image.Image elements)image_crop(crops PIL.Image.Image elements)image_rotate(rotates PIL.Image.Image elements)image_transform(transforms PIL.Image.Image elements)image_brightness(modify brightness of PIL.Image.Image elements)image_contrast(modify contrast of PIL.Image.Image elements)image_filter(apply an image filter to PIL.Image.Image elements)noise(adds noise to the data)center(modify each item according to dataset statistics)normalize(modify each item according to dataset statistics)standardize(modify each item according to dataset statistics)whiten(modify each item according to dataset statistics)randomly(apply data transformations with some probability)Dataset combinationsconcat(concatenate two datasets, placing the items of one after the other)zip(zip datasets itemwise, extending the size of each item)cartesian_product(create a dataset whose items are all combinations of items (zipped) of the originating datasets)