Package: bert-etl
[![Documentation Status](https://readthedocs.org/projects/bert-etl/badge/?version=latest)](https://bert-etl.readthedocs.io/en/latest/?badge=latest)# Bert
A microframework for simple ETL solutions.## ArchitectureAt its core,bert-etluses Dynamodb Streams to communicate between lambda functions.bert-etl.yamlprovides control on how the initial lambda function is called, either by periodic events, sns topics, or s3 bucket (planned)events. Passing an event tobert-etlis straight forward fromzappaor a generic AWS lambda function you’ve hooked up to API Gateway.At this moment in time, there are no plans to attach API Gateway tobert-etl.yamlbecause there is already great software(likezappa) that does this.## Warning: aws-lambda deploy target still considered betabert-etlships with a deploy target toaws-lambda. This feature isn’t very well documented yet, and has quite a bit of work to de done so it may function more consistently. Be aware thataws-lambdais a product ran and controlled by AWS. If you incure charges usingbert-etlwhile utilizingaws-lambda, you may not consider us responsible.bert-etlis offered underMITlicense which includes aUse at your own riskclause.## Begin withLets begin with an example of loading data from a file-server and than loading it into numpy arrays` $ virtualenv-p$(which python3) env $ source env/bin/activate $ pip installbert-etl$ pip install librosa # for demo project $ docker run-p6379:6379-dredis #bert-etlruns on redis to share data across CPUs $bert-runner.py-ndemo $PYTHONPATH='.'bert-runner.py-mdemo-jsync_sounds-f`## Release Notes### 0.3.0Added Error Management. When an error occurs, bert-runner will log the error and re-run the job. If the same error happens often enough, the job will be aborted### 0.2.1Added Release Notes### 0.2.0Added Redis Service auto run. Using docker, redis will be pulled and started in the backgroundAdded Redis Service channels, sometimes you’ll want to run to etl-jobs on the same machine## Fund Bounty Target UpgradesBert provides a boiler plate framework that’ll allow one to write concurrent ETL code using Pythons’microprocessingmodule. One function starts the process, piping data into a Redis backend that’ll then be consumed by the next function. The queues are respectfully named for the scope of the function: Work(start) and Done(end) queue. Please consider contributing to Bert Bounty Targets to improve this documentationhttps://www.patreon.com/jbcurtin## RoadmapCreate configuration file,bert-etl.yamlSupport conda venvSupport pyenv venvSupport dynamodb flushSupport multipule invocations per AWS accountSupport undeploy AWS LambdaSupport Bottle functions in AWS Lambda## Tutorial RoadmapIntroduce Bert APIExplainbert.bindingExplaincomm_binderExplainwork_queueExplaindone_queueExplainologgerExplainDEBUGand how turning it off allows for x-concurrent processesShow an example on how to load timeseries data, calcualte the mean, and display the final output of the meanExpand the example to show how to scale the application implicitlyShow how to run locally using RedisShow how to run locally without Redis, using Dynamodb insteadShow how to run remotly using AWSLambda and DynamodbTalk about dynamodb and eventual consistency