Package: data-plumber
data-plumberdata-plumberis a lightweight but versatile python-framework for multi-stage information processing.
It allows to construct processing pipelines from both atomic building blocks and via recombination of existing pipelines.
Forks enable more complex (i.e. non-linear) orders of execution.
Pipelines can also be collected into arrays that can be executed at once with the same input data.ContentsUsage ExampleInstallDocumentationChangelogUsage exampleConsider a scenario where the contents of a dictionary have to be validated and a suitable error message has to be generated.
Specifically, a valid input-dictionary is expected to have a key "data" with the respective value being a list of integer numbers.
A suitable pipeline might look like this>>> from data_plumber import Stage, Pipeline, Previous
>>> pipeline = Pipeline(
...   Stage(  # validate "data" is passed into run
...     primer=lambda **kwargs: "data" in kwargs,
...     status=lambda primer, **kwargs: 0 if primer else 1,
...     message=lambda primer, **kwargs: "" if primer else "missing argument"
...   ),
...   Stage(  # validate "data" is list
...     requires={Previous: 0},
...     primer=lambda data, **kwargs: isinstance(data, list),
...     status=lambda primer, **kwargs: 0 if primer else 1,
...     message=lambda primer, **kwargs: "" if primer else "bad type"
...   ),
...   Stage(  # validate "data" contains only int
...     requires={Previous: 0},
...     primer=lambda data, **kwargs: all(isinstance(i, int) for i in data),
...     status=lambda primer, **kwargs: 0 if primer else 1,
...     message=lambda primer, **kwargs: "validation success" if primer else "bad type in data"
...   ),
...   exit_on_status=1
... )
>>> pipeline.run().last_message
'missing argument'
>>> pipeline.run(data=1).last_message
'bad type'
>>> pipeline.run(data=[1, "2", 3]).last_message
'bad type in data'
>>> pipeline.run(data=[1, 2, 3]).last_message
'validation success'See section "Examples" inDocumentationfor more explanation.InstallInstall usingpipwithpip install data-plumberConsider installing in a virtual environment.DocumentationOverviewPipelineStageForkStageRefPipelineOutputPipearrayExamplesOverviewBackdata_plumberis designed to provide a framework for flexible data-processing based on re-usable building blocks.At its core stands the classPipelinewhich can be understood as both a container for a collection of instructions (PipelineComponent) and an interface for the execution of a process (Pipeline.run(...)).
Previously definedPipelines can be recombined with otherPipelinesor extended by individualPipelineComponents.
IndividualPipeline.runs can be triggered with run-specific input data.PipelineComponentsare either units defining actual data-processing (Stage) or control the flow of aPipelineexecution (Fork).
Until aForkis encountered, aPipeline.runiterates a pre-configured list ofPipelineComponents.
AnyStage-type component provides an integer status value after execution which is then available for defining conditional execution ofStages or changes in flow (Fork).AStageitself consists of multiple (generally optional) highly customizable sub-stages and properties that can be configured at instantiation.
In particular, a versatile referencing system based on (pre-defined)StageRef-type classes can be used to control the requirements forStages.
Similarly, this system is also used byForks.The output of aPipeline.runis of typePipelineOutput. It contains extensive information on the order of operations, the response from individualStages, and adata-property (of customizable type).
The latter can be used to store and/or output processed data from thePipeline's execution context.Finally, aside from recombination ofPipelines into more complexPipelines, multiple instances ofPipelines can be pooled into aPipearray.
This construct allows to call differentPipelines with identical input data.PipelineBackBuilding anonymous PipelinesAPipelinecan be created in an empty, a partially, or a fully assembled state.For the emptyPipelinea simple expression like>>> from data_plumber import Pipeline, Stage
>>> Pipeline()
<data_plumber.pipeline.Pipeline object at ...>suffices. Following up with statements like>>> p = Pipeline()
>>> p.append(Stage())
>>> p.prepend(Pipeline())
>>> p.insert(Stage(), 1)or simply by using the+-operator>>> p = Pipeline()
>>> Stage() + p + Pipeline()
<data_plumber.pipeline.Pipeline object at ...>Note that when adding to existingPipelines, the change is made in-place.>>> p = Pipeline(Stage())
>>> len(p)
1
>>> p + Stage()
<data_plumber.pipeline.Pipeline object at ...>
>>> len(p)
2Consequently, only properties of the first argument are inherited (refer to python's operator precedence).
Therefore, the use of this operation in combination withPipelines requires caution.Building named PipelinesInstead of simply providing the individualPipelineComponentsas positional arguments during instantiation, they can be assigned names by providing components as keyword arguments (kwargs).
In addition to the kwargs, the positional arguments are still required to determine the order of operations for thePipeline.
These are then given by thePipelineComponent's name:>>> Pipeline(
...   "a", "b", "a", "c",
...   a=Stage(...,),
...   b=Stage(...,),
...   c=Stage(...,)
... )
<data_plumber.pipeline.Pipeline object at ...>In the example above, thePipelineexecutes theStages in the order ofa > b > a > c(note that the names ofStages can occur multiple times in the positional arguments or viaPipeline-extending methods).
Methods likePipeline.appendalso accept string identifiers forPipelineComponents.
If none are provided at instantiation, an internally generated identifier is used.The two approaches of anonymous and namedPipelines can be combined freely:>>> Pipeline(
...   "a", Stage(...,), "a", "c",
...   a=Stage(...,),
...   c=Stage(...,)
... )
<data_plumber.pipeline.Pipeline object at ...>Unpacking PipelinesPipelines support unpacking to be used as, for example, positional or keyword arguments in the constructor of anotherPipeline:>>> p = Pipeline("a", ..., a=Stage(), ...)
>>> Pipeline("b", *p, ..., b=Stage(), **p, ...)
<data_plumber.pipeline.Pipeline object at ...>Running a PipelineAPipelinecan be triggered by calling therun-method.>>> Pipeline(...).run(...)
PipelineOutput(...)Any kwargs passed to this function are forwarded to itsPipelineComponent'sCallables.
Note that some keywords are reserved (out,primer,status,count, andrecords).WhileForks are simply evaluated and their returnedStageRefis used to find the next target for execution,Stages have themselves multiple sub-stages.
First, thePipelinechecks theStage's requirements, then executions itsprimerbefore running theaction-command.
Next, anyexported kwargs are updated in thePipeline.runand, finally, thestatusandresponsemessage is generated (seeStagefor details).Pipeline settingsAPipelinecan be configured with multiple properties at instantiation:initialize_output: aCallablethat returns an object which is consequently passed forward into thePipelineComponent'sCallables; this object is refered to as "persistent data-object" (default generates an empty dictionary)finalize_output: aCallablethat is called before (normally) exiting thePipeline.runwith therun's kwargs as well as the persistent data-objectexit_on_status: either integer value (Pipelineexists normally if any component returns this status) or aCallablethat is called after any component with the component's status (if it evaluates toTrue, thePipeline.runis stopped)loop: boolean; ifFalse, thePipelinestops automatically after iterating beyond the lastPipelineComponentin its list of operations; ifTrue, the execution loops back into the first componentRunning a Pipeline as decoratorAPipelinecan be used to generate kwargs for a function (i.e., based on the content of the persistent data-object).
This requires the data-object being unpackable like a mapping (e.g. a dictionary).>>> @Pipeline(...).run_for_kwargs(...)
... def function(arg1, arg2): ...Arguments that are passed to a call of the decorated function take priority over those generated by thePipeline.run.StageBackAStagerepresents a single building block in the processing logic of aPipeline.
It provides a set ofCallables that are used in aPipeline.run.
Their order of execution is given byprimer>action>export>status>message.
The arguments that are passed into thoseCallables vary byCallableand execution context.
Below a list of all keywords that can occur is given (mostCallables receive only a subset of these):all kwargs given toPipeline.runor exported during execution are forwarded (note that this makes the following arguments reserved words in this context)out: a persistent data-object that is passed through the entirePipeline; its initial value is generated by thePipeline'sinitialize_outputprimer: output ofStage.primerstatus: output ofStage.statuscount: index ofStagein execution ofPipelineStage propertiesStages accept a number of different (optional) arguments that are mostlyCallables to be used by aPipelineduring execution.requires: requirements forStage-execution being eitherNone(always run thisStage) or a dictionary with pairs of references to aStageand the required status (uses most recent evaluation);key types are eitherStageRef,str(identifier of aStagein the context of aPipeline), orint(relative index inPipelinestage arrangement);values are either an integer or aCallabletaking the status as an argument and returning abool(if it evaluates toTrue, theStage-requirement is met)>>> from data_plumber import Pipeline, Stage, Previous
>>> Pipeline(
...   Stage(
...     message=lambda **kwargs: "first stage",
...     status=lambda **kwargs: 1
...   ),
...   Stage(
...     requires={Previous: 0},
...     message=lambda **kwargs: "second stage"
...   ),
... ).run().last_message
'first stage'primer:Callablefor pre-processing data(kwargs:out,count)>>> Pipeline(
...   Stage(
...     primer=lambda **kwargs: "primer value",
...     message=lambda primer, **kwargs: primer
...   ),
... ).run().last_message
'primer value'action:Callablefor main-step of processing(kwargs:out,primer,count)>>> Pipeline(
...   Stage(
...     action=lambda out, **kwargs: out.update({"new_data": 0})
...   ),
... ).run().data
{'new_data': 0}export:Callablethat returns a dictionary of additional kwargs to be exported to the parentPipeline; in the followingStages, these kwargs are then available as if they were provided with thePipeline.run-command(kwargs:out,primer,count)>>> Pipeline(
...   Stage(
...     export=lambda **kwargs: {"new_arg": 0}
...   ),
...   Stage(
...     message=lambda **kwargs:
...       "export successful" if "new_arg" in kwargs
...       else "missing new_arg"
...   ),
... ).run().last_message
'export successful'status:Callablefor generation of aStage's integer exit status(kwargs:out,primer,count)message:Callablefor generation of aStage's exit message(kwargs:out,primer,count,status)ForkBackAForkrepresents a conditional in the execution of aPipeline.
It can be used to redirect the nextPipeline-target to a specific, absolutely or relatively positionedPipelineComponent.
Analogous to theStage, aFork'seval-method is called with a numer of keyword arguments:all kwargs given toPipeline.runare forwarded (note that this makes the following arguments reserved words in this context)out: a persistent data-object that is passed through the entirePipeline; its initial value is generated by thePipeline'sinitialize_outputcount: index ofStagein execution ofPipelinerecords: a list of previously generatedStageRecords (seePipelineOutputfor more information)Fork propertiesAForktakes a singleCallableas argument.
Based on the properties described above, a reference to a targetStageis returned.
This reference can be made as one of several ways:integer; relative index in thePipeline's list of componentsstring; aPipelineComponent's string identifier in the context of aPipeline.runStageRef; a more abstract form of reference, e.g.First,Next(seeStageReffor details)None; signal to (normally) exitPipeline.runExample>>> from data_plumber import Pipeline, Stage, Fork, Next
>>> p = Pipeline(
...   Stage(
...     message=lambda **kwargs: "stage 1 executed"
...   ),
...   Fork(
...     lambda **kwargs: Next if "arg" in kwargs else None
...   ),
...   Stage(
...     message=lambda **kwargs: "stage 2 executed"
...   ),
... )
>>> p.run(arg=0).last_message
'stage 2 executed'
>>> p.run().last_message
'stage 1 executed'StageRefBackStageRefs can be utilized to formulate requirements forStages as well as flow control withForks.
While additional types ofStageRefs can be defined,data_plumberalready provides rich possibilities natively.There are two different categories ofStageRefs:referring to records of previously executedPipelineComponents(a record can uniquely identify a component within thePipeline's sequence of components)referring to a component within the list of registered components of aPipelinedirectlyList of predefined StageRefs (by record)First: record of first componentPrevious: record of previous component (one step)PreviousN(n): record of previous component (nsteps)List of predefined StageRefs (by sequence)Last: last component in sequenceNext: next component in sequence (one step)Skip: component after next in sequence (two steps)NextN(n): next component (nsteps)StageById(id): first occurrence ofidin sequenceStageByIndex(index): component atindexof sequenceStageByIncrement(n): component with relative positionnin sequenceExample>>> from data_plumber import Pipeline, Stage, Fork, Previous, NextN
>>> output = Pipeline(
...   Stage(
...     status=lambda **kwargs: 0
...   ),
...   Stage(
...     requires={Previous: 0},
...     status=lambda count, **kwargs: count
...   ),
...   Fork(
...     lambda count, **kwargs: NextN(1)
...   ),
...   exit_on_status=lambda status: status > 3,
...   loop=True
... ).run()
>>> len(output.records)
6PipelineOutputBackList of propertiesThe output of aPipeline.runis an object of typePipelineOutput.
This object has the following properties:records: a list ofStageRecords corresponding to allStages executed by thePipeline;StageRecords themselves are a collection of propertiesindex: index position inPipeline's sequence ofPipelineComponentsid_: name/id ofStagemessage: the message returned by theStagestatus: the message returned by theStage(for legacy support (<=1.11.) this property can also be indexed, wheremessageandstatusare returned for indices 0 and 1, respectively)kwargs: a dictionary with the keyword arguments used in thePipeline.rundata: the persistent data-object that has been processed through thePipelineFor convenience, the lastStageRecordgenerated in thePipelinecan be investigated using the shortcutslast_record:StageRecordof last component that generated an outputlast_status: status-part of thelast_recordlast_message: message-part of thelast_recordPipearrayBackAPipearrayis a convenience class that offers to run multiplePipelines based on the same input data.
Just like thePipelines themselves, thePipearraycan be either anonymous or named, depending on the use of positional and keyword arguments during setup.
The return type can then be either a list (only positional arguments) or a dictionary with keys being names/ids (at least one namedPipeline). Both contain thePipelineOutputobjects of the individualPipelines.Example>>> from data_plumber import Pipeline, Pipearray
>>> Pipearray(Pipeline(...), Pipeline(...)).run(...)
<list[PipelineOutput]>
>>> Pipearray(
...   p=Pipeline(...),
...   q=Pipeline(...)
... ).run(...)
<dict[str, PipelineOutput]>ExamplesBackInput Processing RevisitedReconsider the introductory example of validating dictionary content.
The described scenario where data is validated against some schema with increasing attention to detail with everyStageand generating helpful response messages is common in the context of, for example, user input.The following section goes into more detail regarding the reasoning for the example-Pipeline's setup.
APipelinegenerally iterates the provided sequence ofStages if there is no flow control included (like in this example).
Within aStagethere is again an order of operations for theCallables as defined in the constructor (primer>action>export>status>message).
Starting with the firstStageof the example,...
...   Stage(  # validate "data" is passed into run
...     primer=lambda **kwargs: "data" in kwargs,
...     status=lambda primer, **kwargs: 0 if primer else 1,
...     message=lambda primer, **kwargs: "" if primer else "missing argument"
...   ),
    ...theprimeris executed first.
This function validates that an argument with keyworddatahas been passed in to thePipeline.run, i.e.Pipeline.run(data=...).
Following this evaluation, the result can be used in the next twoCallables,statusandmessageto generate an output from thisStagewith appropriate properties for the given situation.In the nextStage,...
...   Stage(  # validate "data" is list
...     requires={Previous: 0},
...     primer=lambda data, **kwargs: isinstance(data, list),
...     status=lambda primer, **kwargs: 0 if primer else 1,
...     message=lambda primer, **kwargs: "" if primer else "bad type"
...   ),
    ...the general structure is similar.
Aprimeris executed to validate a property of thedata-keyword argument. This keyword can now be written explicitly into thelambda's signature (instead of covering this argument implicitly with**kwargslike before) since this call toprimerprecedes a requirement-check.
The requirement given here declares only a single requirement, i.e. the previousStage(denoted by theStageRefnamedPrevious) returned with a status of 0.
Consequently, this secondStageis only executed if theprimerof the firstStagereturned a value ofTrue.The thirdStageis conceptually the same as the second one but intended to handle another aspect of the validation.The exit from thePipeline.runcan occur either after completing the thirdStageor if aStagereturns a specific status as defined by the kwarg of the Pipeline itself:>>> pipeline = Pipeline(
...  ...
...   exit_on_status=1
... )This is the reason for returning a status of 1 in allStages if any stage of the validation fails.In the following, two additional features ofdata-plumberare demonstrated by extending the original example.Input Processing - Formatting Output DataAPipelinecan also handle reformatting of input data into a desired format.
Suppose that, following validation, only the sum of all integers given in the listdatais of interest.
In order make this information readily available right after execution within thePipelineOutput-object that is returned by aPipeline.run, one may add a suitableactioncallable in the lastStageof the previous setup:...
...   Stage(  # validate "data" contains only int
...    ...
...     action=lambda data, primer, **kwargs:
...       data.update{"sum": sum(data)} if primer else None
...    ...
...   ),
    ...With this change, after full validation, a keysumis added to the dictionary inPipelineOutput.data, i.e.>>> output = pipeline.run(data=[1, 2, 3])
>>> output.data["sum"]
6Input Processing - Optional ValuesConsider that the data that is fed into thePipeline.run-command is an optional property.
Consequently, it is fine if it is missing but, if it exists, it should satisfy a specific format.This scenario can, for example, be realized using aFork.
AForktakes a singleCallableon construction which returns a reference to anotherStageas next target in the currentPipeline.runor give an exit-signal (by returningNone).
Here, the latter is more interesting.
We can change the initial portion of thePipelinesetup to accomodate for the changes in validation logic:>>> from data_plumber import Next
>>> pipeline = Pipeline(
...   Fork(
...     lambda **kwargs: Next if "data" in kwargs else None
...   )
...   Stage(  # validate "data" is list
...     ...
...   ),
...  ...
...   exit_on_status=1
... )Now, the validation of the firstStagein the initial example is replaced with aFork.Nextis another pre-definedStageRefpointing at the nextPipelineComponentin thePipeline's sequence.Note that for this setup, ifdatais missing, the returnedPipelineOutputdoes not contain anyStageRecords asForkdoes not output any status.
In order to have both, theFork-logic and aStageRecord, theForkhas to be the second component.
Using thestages-kwarg for theFork'sCallableenables to use the status from a previousStageand circumvent a duplicate validation.Changelog[1.12.1] - 2024-02-07Fixedupdate and fix errors in documentation (2318210)[1.12.0] - 2024-02-05Changedadded a list of previousStageRecords as kwarg for the call to aFork's conditional (2f1cb77)changedStageRecordinto a proper dataclass (e7eae6d)[1.11.0] - 2024-02-04Changedadded common base class_PipelineComponentforPipelinecomponentsStageandFork(f628159)Addedadded docs to package metadata (061e311)names forPipelineComponentscan now be declared in extension methods (append,prepend, ...) (8363284)Pipelinenow supportsin-operator (usable with either component directly or its name/id) (5701073)added requirements forPipelineto be unpacked as mapping (b2db8fa)Fixedfixed issue whereFork-objects were internally not registered by their id (b267ca4)[1.8.0] - 2024-02-03ChangedrefactoredForkandStageto transform string/integer-references toStages intoStageRefs (7ba677b)Addedadded decorator-factoryPipeline.run_for_kwargsto generate kwargs for function calls (fe616b2)added optionalStage-callable to export kwargs intoPipeline.run(8eca1bc)added even more types ofStageRefs:PreviousN,NextN(576820c)addedpy.typed-marker to package (04a2e1d)added more types ofStageRefs:StageById,StageByIndex,StageByIncrement(92d57ad)[1.4.0] - 2024-02-01Changedrefactored internal modules (cf7045f)AddedaddedStageRefsNext,Last, andSkip(14abaa7)added optional finalizer-CallabletoPipeline(d95e5b6)added support forCallableinPipeline-argumentexit_on_status(154c67b)FixedPipelineOutput.last_X-methods now returnNonein case of empty records (b7a6ba1)[1.0.0] - 2024-01-31ChangedBreaking:refactorPipelineOutputand related types (1436ca1)Breaking:replaced forwarding kwargs ofPipeline.runas dictionaryin_intoStage/Fork-Callables by forwarding directly (f2710fa,b569bb9)Addedadded missing information in module- and class-docstrings (7896742)[0.1.0] - 2024-01-31initial release