Package: graphtik
(build-version: x.x.x, build-date: 2023-04-25T21:27:33.616654)It’s a DAG all the way down!Computation graphs for Python & PandasGraphtikis a library to compose, solve, execute & plotgraphs of python functions(a.k.a pipelines) that consume and populate named data
(a.k.a dependencies), whose names may be nested (such as.pandasdataframe columns),
based on whether values for those dependencies exist in the inputs or
have been calculated earlier.In mathematical terms, given:a partially populated data-tree, anda set of functions operating on (consuming/producing) branches of the data tree,graphtikcollects a subset of functions in a graph that when executed
consume & produce as many values as possible in the data-tree.Its primary use case is building flexible algorithms for data science/machine learning projects.It should be extendable to implement the following:anIoC dependency resolver(e.g. Java Spring, Google Guice);an executor of interdependent tasks based on files (e.g. GNU Make);a custom ETL engine;a spreadsheet calculation engine.GraphtiksprangfromGraphkit(summer 2019, v1.2.2) toexperimentwith Python 3.6+ features,
but has diverged significantly with enhancements ever since.FeaturesDeterministic pre-decidedexecution plan(unlesspartial-outputsorendured operationsdefined, see below).Can assemble existing functions without modifications intopipelines.dependencyresolution can bypass calculation cycles based on data given and asked.Support functions withoptional <optionals>input args and/orvarargs <varargish>.Support functions withpartial outputs; keep working even if certainenduredoperations fail.Facilitate trivialconveyor operations andaliasonprovides.Support cycles, by annotating repeated updates ofdependencyvalues assideffects,
(e.g. to add columns intopandas.DataFrames).Hierarchical dependencies <subdoc>may access data values deep insolutionwithjson pointer pathexpressions.Hierarchical dependencies annotated asimplicitimply which subdoc dependency
the function reads or writes in the parent-doc.Merge <operation merging>ornest <operation nesting>sub-pipelines.Earlyevictionof intermediate results fromsolution, to optimize memory footprint.Solution tracks all intermediateoverwritten <overwrite>values for the same dependency.ElaborateGraphvizplotting with configurableplot themes.Integration with Sphinx sites with the newgraphtikdirective.Authored withdebuggingin mind.Parallel execution (but underdeveloped & DEPRECATED).Anti-featuresIt’s not meant to follow complex conditional logic based ondependencyvalues
(though it does support that toa limited degree <partial outputs>).It’s not an orchestrator for long-running tasks, nor a calendar scheduler -Apache Airflow,DagsterorLuigimay help for that.It’s not really a parallelizing optimizer, neither a map-reduce framework - look
additionally atDask,IpyParallel,Celery,
Hive, Pig, Hadoop, etc.It’s not a stream/batch processor, like Spark, Storm, Fink, Kinesis,
because it pertains function-call semantics, calling only once each function
to process data-items.Differences withschedulaschedulais a powerful library written roughly
for the same purpose, and works differently along these lines
(ie features below refer toschedula):terminology (<graphtik> := <schedula>):pipeline := dispatcherplan := workflowsolution := solutionDijkstra planning runs while calling operations:Powerful & flexible (ie all operations are dynamic,domainsare possible, etc).Supportsweights.Cannot pre-calculate & cache execution plans (slow).Calculated values are stored inside a graph (mimicking the structure of the functions):graph visualizations absolutely needed to inspect & debug its solutions.graphs imply complex pre/post processing & traversal algos(vs constructing/traversing data-trees).Reactive plotted diagrams, web-server runs behind the scenes.Operation graphs are stackable:plotted nested-graphs support drill-down.graphtikemulates that with data/operation names (operation nesting),
but always a unified graph is solved at once,
bc it is impossible to dressnesting-funcsas apython-funcsand pre-solve plan
(scheduladoes not pre-solve plan, Dijkstra runs all the time).
See TODO about plotting such nested graphs.Scheduladoes not calculate all possible values (ie nooverwrites).Schedulacomputes precedence based on weights and lexicographical order of function name.Re-inserting operation does not overrides its current function - must remove it first.graphtikprecedence based insertion order duringcomposition.Virtualstartandenddata-nodes needed for Dijkstra to solve the dag.No domains (execute-time conditionals deciding whether a function must run).Probablyrecomputeis more straightforward ingraphtik.TODO: more differences withschedulaexist.Quick startHere’s how to install:pip install graphtikOR with various “extras” dependencies, such as, for plotting:pip install graphtik[plot]. Tip::Supported extras:plotfor plotting withGraphviz,matplotfor plotting inmaplotlibwindowssphinxfor embedding plots insphinx-generated sites,testfor runningpytests,dillmay help for picklingparalleltasks - seemarshallingterm
andset_marshal_tasks()configuration.allall of the above, plus development libraries, egblackformatter.devlikeallLet’s build agraphtikcomputation graph that produces x3 outputs
out of 2 inputsαandβ:α x βα - αxβ|α - αxβ| ^ 3>>> from graphtik import compose, operation
>>> from operator import mul, sub>>> @operation(name="abs qubed",
...            needs=["α-α×β"],
...            provides=["|α-α×β|³"])
... def abs_qubed(a):
...     return abs(a) ** 3Compose theabs_qubedfunction along themul&subbuilt-ins
into a computation graph:>>> graphop = compose("graphop",
...     operation(needs=["α", "β"], provides=["α×β"])(mul),
...     operation(needs=["α", "α×β"], provides=["α-α×β"])(sub),
...     abs_qubed,
... )
>>> graphop
Pipeline('graphop', needs=['α', 'β', 'α×β', 'α-α×β'],
                    provides=['α×β', 'α-α×β', '|α-α×β|³'],
                    x3 ops: mul, sub, abs qubed)Run the graph and request all of the outputs
(notice that unicode characters work also as Python identifiers):>>> graphop(α=2, β=5)
{'α': 2, 'β': 5, 'α×β': 10, 'α-α×β': -8, '|α-α×β|³': 512}… or request a subset of outputs:>>> solution = graphop.compute({'α': 2, 'β': 5}, outputs=["α-α×β"])
>>> solution
{'α-α×β': -8}… and plot the results (if injupyter, no need to create the file):>>> solution.plot('executed_3ops.svg')  # doctest: +SKIP