Package: etlrules
etlrulesA python rule engine for applying transformations to dataframes.ETL stands forExtract, Trasform, Load, which is a three step
process to source the data from some data source (Extract), transform the data (Transform) and publish it to a final
destination (Load).Data transformation of tabular sets can be done in pure python with many dedicated python packages, the most widely
recognized beingpandas. The result of such transformations can be quite opaque with the
logic difficult to read and understand, especially by non-coders. Even coders can struggle to understand certain
transformations unless in-code documentation is added and even when documentation is available, the code change in ways
which renders the documentation stale.The etlrules package solves this by offering a set of simple rules which users can use to form a plan. The plan is a blueprint
on how to transform the data. The plan can be saved to a yaml file, stored in a repo for version control or in a database for
manipulation via UIs and then executed in a repeatable and predictable fashion. The rules in the plan can have names and
extensive description acting as an embedded documentation as to what the rule is trying to achieve.This data-driven way of operating on tabular data allows non-technical users to come up with data transformations which can
automate various problems without the need to code. Workflows for managing change and version control can be put in place
around the plans, allowing technical and non-technical users to collaborate on data transformations that can be scheduled to
run periodically for solving real business problems.High level conceptsPlanA plan is a blueprint of how to perform extractions of tabular data, transformations of the data and how to load (ie write)
the transformed data to its final destination.A plan is a collection of rules, each of which operate on a dataframe (tabular data).RuleA rule is an operation performed on a dataframe. There are three types of rules:
* Extract rules (aka read rules)
- They will read an external data source (ie files, DBs, APIs endpoints) and bring the data into memory for processing
* Transform rules
- They will perform a transformation of the data (ie add a new column, modify an existing column, join columns, aggregate)
* Load rules (aka write rules)
- They will write the output into an external storage (ie files, DBs, APIs endpoints)Rule engineThe component that takes a plan and executes it (rule by rule) based on an input.Rule dataThe structure that holds together any input dataframes, temporary results and the final output of a rule engine execution of a plan.
The rule data can have some input dataframes or they can start as empty canvases, with the plan performing extractions/reading of data that it needs.BackendThe underlying dataframe library to use for executing the plan. For example: pandas, vaex, polars, etc.
At the moment, only pandas is supported.RunnerA command line interface application which can take a plan serialized as a yaml file and run it. Also allows to run that
programmatically via the run_plan import.Documentationhttps://ciprianmiclaus.github.io/etlrules/LicenseFree software: MIT