Package: pipelime
Data Pipeline 101InstallationpipinstallpipelimeBasic UsageUnderfolder FormatTheUnderfolderformat is one of the pipelime dataset formats: i.e. a flexible way to
(model and) store a generic dataset throughfilesystem.An Underfolderdatasetis a collection of samples. Asampleis a collection of items.
Anitemis a unitary block of data, it can be a generic tensor (e.g. a multi-channel image,
or a plain matrix), a dictionary. or more.Underfolder datasets must contain a subfolder nameddatathat will actually contain the
samples and items. Optionally you can store the items in the root folder directly, they
will act as “global” items injected into each sample.Items are named using the following naming convention:$ID_$ITEM.$EXTWhere:$IDis the sample identifier, must be a unique string for each sample.ITEMis the item name.EXTis the item extension. Currently supported extensions are:The most common image formats like PNG, JPEG, BMP, and many others…YAML and JSON for dictionary-like objects.TXT for numpy 2D matrix notation.NPY and NPZ for numpy arrays.PKL for generic pickable python objects.Root files follow the same convention but they lack the sample identifier part:$ITEM.$EXTReading an Underfolder DatasetPipelime provides an intuitive interface to read, manipulate and write Underfolder Datasets.
You don’t have to memorize complex signatures, instantiate weird object iterators, or write
tens of lines of boilerplate code. It all boils down to areader, awriterand objects that
behave like built-in python types such aslistsanddictionaries.frompipelime.sequences.readers.filesystemimportUnderfolderReader# Read an underfolder dataset with a single line of codedataset=UnderfolderReader('tests/sample_data/datasets/underfolder_minimnist')# A dataset behaves like a Sequencelen(dataset)# The number of samples (20)sample=dataset[4]# Get a Sample from the dataset# A Sample is a MutableMappinglen(sample)# The number of items (10)set(sample.keys())# The set of all the item names {'cfg', 'image', 'image_mask', ...}item=sample['image']# Get an item from the sample# An item can be any python object, depending on which extension is used to store it.type(item)# numpy.ndarrayitem.shape# (28, 28, 3)Writing an Underfolder DatasetYou canwritea dataset by simply creating and running a writer object.frompipelime.sequences.writers.filesystemimportUnderfolderWriter# Create the writer object from a destination pathwriter=UnderfolderWriter('/tmp/my_output_dataset')# Write the dataset to file systemwriter(dataset)By default,UnderfolderWritersaves every sample with the extension it was originally read with.
If for any reason it is unable to retrieve the original extension, it will usepickleto
serialize the object.If you don’t want to use pickle, you can choose acustom extensionfor each item name.
You can also choose which items are going to be saved asroot files(if the contained data
is the same for all samples).frompipelime.sequences.writers.filesystemimportUnderfolderWriter# These items are going to be saved as root filesroot_files=['cfg','numbers','pose']# Associate a custom extension to each item nameextensions={'image':'jpg','image_mask':'png','image_maskinv':'png','label':'txt','metadata':'json','metadatay':'yml','points':'txt','numbers':'txt','pose':'txt','cfg':'yml'}# Create a customized writer objectwriter=UnderfolderWriter('/tmp/my_output_dataset',root_files_keys=root_files,extensions_map=extensions,)# Write the dataset to file systemwriter(dataset)