Package: pipelime-python
🍋pipelimeIf life gives you lemons, usepipelime.Welcome topipelime, a swiss army knife for data processing!pipelimeis a full-fledgeframeworkfordata science: read your datasets,
manipulate them, write back to disk or upload to a remote data lake.
Then build up yourdataflowwith Piper and manage the configuration with Choixe.
Finally,embedyour custom commands into thepipelimeworkspace, to act both as dataflow nodes and advanced command line interface.Maybe too much for you? No worries,pipelimeismodularand you can just take out what you need:data processing scripts: use the powerfulSamplesSequenceand create your own data processing pipelines, with a simple and intuitive API. Parallelization works out-of-the-box and, moreover, you can easily serialize your pipelines to yaml/json. Integrations with popular frameworks, e.g.,pytorch, are also provided.easy dataflow:Pipercan manage and execute directed acyclic graphs (DAGs), giving back feedback on the progress through sockets or custom callbacks.configuration management:Choixeis a simple and intuitive mini scripting language designed to ease the creation of configuration files with the help of variables, symbol importing, for loops, switch statements, parameter sweeps and more.command line interface:pipelimecan remove all the boilerplate code needed to create a beautiful CLI for you scripts and packages. You focus onwhat mattersand we provide input parsing, advanced interfaces for complex arguments, automatic help generation, configuration management. Also, anyPipelimeCommandcan be used as a node in a dataflow for free!pydantic tools: most of the classes inpipelimederive frompydantic.BaseModel, so we have built some useful tools to, e.g., inspect their structure, auto-generate human-friendly documentation and more (including a TUI to help you writing input data todeserializeany pydantic model).InstallationInstallpipelimeusing pip:pip install pipelime-pythonTo be able todrawthe dataflow graphs, you need thedrawvariant:pip install pipelime-python[draw]WarningThedrawvariant needsGraphvizhttps://www.graphviz.org/installed on your system
On Linux Ubuntu/Debian, you can install it with:sudo apt-get install graphviz graphviz-devAlternatively you can usecondaconda install --channel conda-forge pygraphvizPlease see the full options athttps://github.com/pygraphviz/pygraphviz/blob/main/INSTALL.txtBasic UsageUnderfolder FormatTheUnderfolderformat is the preferredpipelimedataset formats, i.e., a flexible way to
model and store a generic dataset throughfilesystem.An Underfolderdatasetis a collection of samples. Asampleis a collection of items.
Anitemis a unitary block of data, i.e., a multi-channel image, a python object,
a dictionary and more.
Any valid underfolder dataset must contain a subfolder nameddatawith samples
and items. Also,global shareditems can be stored in the root folder.Items are named using the following naming convention:Where:$IDis the sample index, must be a unique integer for each sample.ITEMis the item name.EXTis the item extension.We currently support many common file formats and others can be added by users:.png,.jpeg/.jpg/.jfif/.jpe,.bmpfor images.tiff/.tiffor multi-page images and multi-dimensional numpy arrays.yaml/.yml,.jsonand.toml/.tmlfor metadata.txtfor numpy 2D matrix notation.npyfor general numpy arrays.pkl/.picklefor picklable python objects.binfor generic binary dataRoot files follow the same convention but they lack the sample identifier part, i.e.,$ITEM.$EXTReading an Underfolder Datasetpipelime provides an intuitive interface to read, manipulate and write Underfolder Datasets.
No complex signatures, weird object iterators, or boilerplate code, you just need aSamplesSequence:frompipelime.sequencesimportSamplesSequence# Read an underfolder dataset with a single line of codedataset=SamplesSequence.from_underfolder('tests/sample_data/datasets/underfolder_minimnist')# A dataset behaves like a Sequenceprint(len(dataset))# the number of samplessample=dataset[4]# get the fifth sample# A sample is a mappingprint(len(sample))# the number of itemsprint(set(sample.keys()))# the items' keys# An item is an object wrapping the actual dataimage_item=sample["image"]# get the "image" item from the sampleprint(type(image_item))# <class 'pipelime.items.image_item.PngImageItem'>image=image_item()# actually loads the data from disk (may have been on the cloud as well)print(type(image))# <class 'numpy.ndarray'>Writing an Underfolder DatasetYou canwritea dataset by calling the associated operation:# Attach a "write" operation to the datasetdataset=dataset.to_underfolder('/tmp/my_output_dataset')# Now run over all the samplesdataset.run()# You can easily spawn multiple processes if neededdataset.run(num_workers=4)