Package: eruptr
Don't ETL or ELT. LET your data be free.VulknData EruptrData needs a Makefile.https://tonsky.me/blog/disenchantment/Eruptr is an extensible, model and configuration driven data management system
targeting ETL, DataOps and Analytics Engineering.The initial target for Eruptr is the ClickHouse OLAP engine however it is being
adapted and extended to address other modern data eco-systems.WhyThe main data processing mechanism is over untyped data using existing decades
old Unix/Linux tools. One of the key goals of Eruptr in this regard is to force
us to ask honest questions about existing data processing mechanisms. Do we need
a heavy, Java-based system to process our data or does awk solve the problem for
95% of our data in 10% of the time? What about heavily typed data processing
systems in Python that pass dictionaries or tuples around with mountains of code
on Kubernetes? Is that necessary? Or could it be dealt with via grep on a micro
instance?Initial ReleaseThe initial release is an alpha release. Tests have not been established and
there are quirks/issues in some components or functionality that differs to the
documentation. These issues will be rectified prior to Christmas 2020 with
a proper production release in January 2021.RoadmapGenerator modules with daemon modeScheduler/task controller with multi-processing schedulersEngine support for PostgreSQL, MySQL, RedShift and Google BigQueryAdditional task, pipe and io modulesAnalytics / Data Engineering projectsDatabase reflection with automatic model/schema migrationsA web UI for managing workflowsAnalytics engineering workflows including documentationGraph scheduling with multiple flowsMarrying eco-systems - Vulkn and the upcoming Vulkn ServerDocumentationDocumentation is available athttp://docs.vulkndata.io/eruptr/.InstallationEruptr has only been tested with recent versions of Ubuntu and Python 3.7. You
will need to have a working Python 3.7.x environment with pip.Ensure you have installed ClickHouse the clickhouse-client program including
clickhouse-local:sudoaptinstallclickhouse-clientclickhouse-commonInstallation with pipInstall Eruptr via pip.sudoadd-apt-repositoryppa:deadsnakes/ppa
sudoaptupdate
sudoaptinstall-ypython3.7python3.7-devpython3-pip
sudopython3.7-mpipinstalleruptrInstallation from source (for developers)Install Eruptr viagit clone.gitclonehttps://github.com/VulknData/eruptr.gitcderuptrInstall required packages. Note that it may make sense to do this within a
virtual environment.pipinstall-rrequirements.txtYou can start using Eruptr via the eruptr script:cdscriptssourceenv.sh
./eruptr--helpGetting StartedLet's use one of the provided examples. This assumes you have a ClickHouse
server running on your local system listening on standard ports with no default
credentials (default/empty password).Create the following file - simple1.yaml:name:Simple1 Batch Jobshard:clickhouse://localhost/testworkflow:-drop:executor:StepExecutor-create:executor:StepExecutorenabled:true-data:executor:StepExecutor-pre:executor:StepExecutorenabled:true-input:executor:UnixPipeExecutorenabled:true-post:executor:StepExecutor-clean:executor:StepExecutordata:-tasks.file.write:path:system_metrics.csvdata:|"device1","1970-04-27 03:46:40",32.3,"temperature""device1","1970-04-27 03:46:40",10.2,"watts"-tasks.pack.gz:input_file:system_metrics.csvoutput_file:system_metrics.csv.gzpre:-TRUNCATE TABLE simple1.system_metricsinput:-io.file.read:system_metrics.csv.gz-pipes.unpack.gz-io.clickhouse.write:table:simple1.system_metricsformat:formats.clickhouse.CSVpost:-tasks.file.delete:system_metrics.csv.gzcreate:-CREATE DATABASE IF NOT EXISTS simple1-|CREATE TABLE IF NOT EXISTS simple1.system_metrics(device String,epoch_dt DateTime,value Float32,tag String) ENGINE = Logclean:-DROP TABLE IF EXISTS simple1.system_metrics-DROP DATABASE IF EXISTS simple1First lets generate some test data:eruptrload--confsimple1.yaml--log-levelINFO--flowsdata

VulknDataEruptr(C)2020VulknData,JasonGodden

GPLv3-seehttps://github.com/VulknData/eruptr/COPYING12/01/202009:14:40PM-INFO-Renderingconfiguration12/01/202009:14:40PM-INFO-Running"Simple1 Batch Job"12/01/202009:14:40PM-INFO-Executingdatasection12/01/202009:14:40PM-INFO-StepExecutor:tasks.file.write({'path':'system_metrics.csv','data':'"device1","1970-04-27 03:46:40",32.3,"temperature"\n"device1","1970-04-27 03:46:40",10.2,"watts"\n'})->tasks.pack.gz({'input_file':'system_metrics.csv','output_file':'system_metrics.csv.gz'})12/01/202009:14:40PM-INFO-Successfullycompleted"Simple1 Batch Job"OK-Simple1BatchJob-SUCCESSUsing the --flows option we've told eruptr to execute thedataflow only.Great. This has provided us with a simple dataset we can import. If we don't
specify any flows eruptr automatically runs only the enabled workflows in the
order they're defined (create, pre and input).eruptrload--confsimple1.yaml--log-levelINFO

VulknDataEruptr(C)2020VulknData,JasonGodden

GPLv3-seehttps://github.com/VulknData/eruptr/COPYING12/01/202009:24:04PM-INFO-Renderingconfiguration12/01/202009:24:04PM-INFO-Running"Simple1 Batch Job"12/01/202009:24:04PM-INFO-Executingcreatesection12/01/202009:24:04PM-INFO-StepExecutor:tasks.clickhouse.execute(CREATEDATABASEIFNOTEXISTSsimple1)->tasks.clickhouse.execute(CREATETABLEIFNOTEXISTSsimple1.system_metrics(deviceString,epoch_dtDateTime,valueFloat32,tagString)ENGINE=Log)12/01/202009:24:04PM-INFO-Executingpresection12/01/202009:24:04PM-INFO-StepExecutor:tasks.clickhouse.execute(TRUNCATETABLEsimple1.system_metrics)12/01/202009:24:04PM-INFO-Executinginputsection12/01/202009:24:04PM-INFO-UnixPipeExecutor:<functionreadat0x7f9e0d61ed90>(run='system_metrics.csv.gz',connection='clickhouse://localhost/test'|<function<lambda>at0x7f9e0d62bd08>(run='None',connection='clickhouse://localhost/test'|<functionwriteat0x7f9e0d61ed08>(connection='clickhouse://localhost/test',table='simple1.system_metrics',format='formats.clickhouse.CSV'12/01/202009:24:04PM-INFO-Successfullycompleted"Simple1 Batch Job"OK-Simple1BatchJob-SUCCESSAnd what can we see in ClickHouse?hulk:)select*fromsimple1.system_metricsformatCSV;SELECT*
FROMsimple1.system_metrics
FORMATCSV

Queryid:f4c471b1-f32a-4636-8e62-d923e5dc10c4"device1","1970-04-27 03:46:40",32.3,"temperature""device1","1970-04-27 03:46:40",10.2,"watts"2rowsinset.Elapsed:0.005sec.Perfect. So eruptr has created the necessary database, then table(s) and then
extracted and loaded our data.This is a trivial example though. Explore the documentation further to see how
you can create complex workflows with everything from custom shell commands to
embedded Python all driven by simple YAML. Or combine Mako and Jinja to create
re-usable macros for processing your data.