Package: judah
judahShe (Leah) said, “This time I will praise the LORD”; so she named him Judah - Genesis 29: 35judah is a service-oriented Python package to handle ETL (extract-transform-load) tasks easily.It follows a service-oriented architectural (SOA) design.Under the hood, it uses the nice little ETL framework calledBonobounder the hood.This project is still under heavy developmentPurposeThe judah framework was created to standardize the integration or ETL (Extract-transform-load) applications that collect
energy data from multiple external sources and saves it in a warehouse.LinksHere are a few important links:RepositoryDocumentationLanguages UsedPython 3.6DependenciesPython3.6(attempting to use > 3.6 may cause weird errors)Bonobo ETLSqlAlchemySeleniumrequestsxml-streamwebdriver-managerxlrdpython-dotenvpydanticemail-notifierGetting StartedInstall the packagepipinstalljudahCopy the.example.envfile to.envand make appropriate edits on itcp.example.env.envImport the source, destination and transformer classes, as well as any utility functions you may like and use accordinglyfromjudah.sources.export_site.date_basedimportDateBasedExportSiteSource# ...Expected App System Design and ArchitectureThe judah framework expects all applications that use it to follow a service-oriented-architecture as shown below.The app should have aservicesfolder (or in python, what we call package) to contain the separate ETL services,
each corresponding to a given third-party data source e.g. CNN, BBCSubsequently, each ETL service should be divided up into child services. Each child service should represent a unique data flow path e.g. REST-API-to-database,
REST-API-to-cache, REST-API-to-queue, file-download-site-to-database, file-download-site-to-queue etc.Each child service should be divided up into a number of microservices.
Each microservice should correspond to a single dataset, e.g. 'available_capacity', 'installed_capacity' etc.Each microservice is expected to have adestinationfolder, asource.pyfile, acontroller.pyfile and atransformers.pyfile.Thedestinationfolder contains the database model file to which the data is to be saved.
It contains a child class of the DatabaseBaseModel class of thejudah frameworkThesource.pyfile contains a child class of the BaseSource class of thejudah framework.
This is the class responsible for connecting to the data source (e.g. the REST API) and downloading the data from there.Thetransfomers.pyfile contains child classes of the BaseTransformer class of thejudah framework.
They are responsible for transforming the source data into the data that can be saved. This may involve changing field names and types, exploading the data etc.Thecontroller.pyfile contains child class of the BaseController class of thejudah framework.
This class is responsible for controlling the data flow from the source class, through the transformers, to the destination model.Each child service foldershould contain a registry of these microservices in its__init__.pyfile. The registry is just a list of the controllers of the microservices.The app should have amain.pyfile as the entry point of the app where theBonobograph is instantiated
and the microservice registries mentioned in the point above are added to the graph. Look at theexample_main.pyfile for inspiration.Why service-oriented architectural (SOA) designService oriented architecture makes it easy to connect actual feature requests with the actual code that is written.
Many a time, software requirements are structured in typically a service-oriented manner.
For example.User can see realtime data about bitcoinUser can see realtime data about EthereumUser can view historical data about bitcoinWhen we have source code that follows the exact manner these requirements are laid out, it is easy to comprehend for
anyone really.For example, for the above example, each of those requirements will have a single pipeline, each having its own
independent folder.It is even easy to transfer that architecture into a stable microservice architecture if there is ever need to do so.Watchthis talk by Alexandra Noonanandthis other one by Simon BrownHow to set up Debian server for Selenium Chrome driverInstall an in-memory display server (xvfb)sudoapt-getupdate
sudoapt-getinstall-ycurlunzipxvfblibxi6libgconf-2-4Install Google Chromesudocurl-sS-o-https://dl-ssl.google.com/linux/linux_signing_key.pub|sudoapt-keyadd-
sudoecho"deb [arch=amd64]  http://dl.google.com/linux/chrome/deb/ stable main">>/etc/apt/sources.list.d/google-chrome.list
sudoapt-get-yupdate
sudoapt-get-yinstallgoogle-chrome-stableHow to testInstall PostgreSQL +9.5  server if you haven't already.Here are the instructionsClone the repo and enter its root foldergitclonehttps://github.com/sopherapps/judah.git&&cdjudahCopy the.example.envfile to.envand make appropriate edits on itcp.example.env.envCreate the test database: 'test_judah' in this casesudo-supostgres
createdbtest_judahUpdate theTEST_POSTGRES_DB_URIvariable in the.envfile to that test database's connection detailsCreate a virtual environment and activate itvirtualenv-p/usr/bin/python3.6env&&sourceenv/bin/activateInstall the dependenciespipinstall-rrequirements.txtRun the test commandpython-munittestTo view test coverage and then report the resultscoveragerun-munittest&&coveragereport-mHow to Use (Example commands for Linux)Ensure you have Google Chrome installed. For debian servers, see instructions under the
title "How to set up Debian server for Selenium Chrome driver"MaintainersMartin AhinduraFolder StructureThejudahpackage holds the framework components that are basically base classes to be overridden.The folder structure as generated by th commandtree -d --matchdirs -I 'env|__pycache__'is as shown below.
├── judah
│   ├── controllers
│   │   ├── base
│   │   ├── db_to_db
│   │   ├── export_site_to_db
│   │   └── rest_api_to_db
│   ├── destinations
│   │   └── database
│   ├── sources
│   │   ├── base
│   │   ├── database
│   │   ├── export_site
│   │   └── rest_api
│   ├── transformers
│   └── utils
└── test
    ├── assets
    ├── test_controllers
    ├── test_destinations
    │   └── test_database
    ├── test_sources
    │   ├── test_database
    │   ├── test_exports_site
    │   └── test_rest_api
    ├── test_transformers
    └── test_utilsAcknowledgementsThe tutorialHow to Setup Selenium with ChromeDriver on Debian 10/9/8was very useful when deploying the app
on a Debian serverTheRealPython tutorialon publishing python packages was very helpful.TheStackoverflow question about a wheel errorwas very helpful.