Package: mara-pipelines
Mara PipelinesThis package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. It has a number of baked-in assumptions/ principles:Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code.PostgreSQL as a data processing engine.Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines.GNU make semantics. Nodes depend on the completion of upstream nodes. No data dependencies or data flows.No in-app data processing: command line tools as the main tool for interacting with databases and data.Single machine pipeline execution based on Python'smultiprocessing. No need for distributed task queues. Easy debugging and output logging.Cost based priority queues: nodes with higher cost (based on recorded run times) are run first.InstallationTo use the library directly, use pip:pip install mara-pipelinesorpip install git+https://github.com/mara/mara-pipelines.gitFor an example of an integration into a flask application, have a look at themara example project 1andmara example project 2.Due to the heavy use of forking, Mara Pipelines does not run natively on Windows. If you want to run it on Windows, then please use Docker or theWindows Subsystem for Linux.ExampleHere is a pipeline "demo" consisting of three nodes that depend on each other: the taskping_localhost, the pipelinesub_pipelineand the tasksleep:frommara_pipelines.commands.bashimportRunBashfrommara_pipelines.pipelinesimportPipeline,Taskfrommara_pipelines.cliimportrun_pipeline,run_interactivelypipeline=Pipeline(id='demo',description='A small pipeline that demonstrates the interplay between pipelines, tasks and commands')pipeline.add(Task(id='ping_localhost',description='Pings localhost',commands=[RunBash('ping -c 3 localhost')]))sub_pipeline=Pipeline(id='sub_pipeline',description='Pings a number of hosts')forhostin['google','amazon','facebook']:sub_pipeline.add(Task(id=f'ping_{host}',description=f'Pings{host}',commands=[RunBash(f'ping -c 3{host}.com')]))sub_pipeline.add_dependency('ping_amazon','ping_facebook')sub_pipeline.add(Task(id='ping_foo',description='Pings foo',commands=[RunBash('ping foo')]),['ping_amazon'])pipeline.add(sub_pipeline,['ping_localhost'])pipeline.add(Task(id='sleep',description='Sleeps for 2 seconds',commands=[RunBash('sleep 2')]),['sub_pipeline'])Tasks contain lists of commands, which do the actual work (in this case running bash commands that ping various hosts).In order to run the pipeline, a PostgreSQL database is recommended to be configured for storing run-time information, run output and status of incremental processing:importmara_db.auto_migrationimportmara_db.configimportmara_db.dbsmara_db.config.databases\=lambda:{'mara':mara_db.dbs.PostgreSQLDB(host='localhost',user='root',database='example_etl_mara')}mara_db.auto_migration.auto_discover_models_and_migrate()Given that PostgresSQL is running and the credentials work, the output looks like this (a database with a number of tables is created):Created database "postgresql+psycopg2://root@localhost/example_etl_mara"

CREATE TABLE data_integration_file_dependency (
    node_path TEXT[] NOT NULL,
    dependency_type VARCHAR NOT NULL,
    hash VARCHAR,
    timestamp TIMESTAMP WITHOUT TIME ZONE,
    PRIMARY KEY (node_path, dependency_type)
);

.. more tablesCLI UIThis runs a pipeline with output to stdout:frommara_pipelines.cliimportrun_pipelinerun_pipeline(pipeline)And this runs a single node of pipelinesub_pipelinetogether with all the nodes that it depends on:run_pipeline(sub_pipeline,nodes=[sub_pipeline.nodes['ping_amazon']],with_upstreams=True)And finally, there is some sort of menu based onpythondialogthat allows to navigate and run pipelines like this:frommara_pipelines.cliimportrun_interactivelyrun_interactively()Web UIMore importantly, this package provides an extensive web interface. It can be easily integrated into anyFlaskbased app and themara example projectdemonstrates how to do this usingmara-app.For each pipeline, there is a page that showsa graph of all child nodes and the dependencies between thema chart of the overal run time of the pipeline and it's most expensive nodes over the last 30 days (configurable)a table of all the pipeline's nodes with their average run times and the resulting queuing priorityoutput and timeline for the last runs of the pipelineFor each task, there is a page showingthe upstreams and downstreams of the task in the pipelinethe run times of the task in the last 30 daysall commands of the taskoutput of the last runs of the taskPipelines and tasks can be run from the web ui directly, which is probably one of the main features of this package:Getting startedDocumentation is currently work in progress. Please use themara example project 1andmara example project 2as a reference for getting started.LinksDocumentation:https://mara-pipelines.readthedocs.io/Changes:https://mara-pipelines.readthedocs.io/en/latest/changes.htmlPyPI Releases:https://pypi.org/project/mara-pipelines/Source Code:https://github.com/mara/mara-pipelinesIssue Tracker:https://github.com/mara/mara-pipelines/issues