Package: keanu-etl
Project KeanuWelcome to Project Keanu!Keanu is an ETL tool to run the queries, and to work on your data transform. It
lets you define an ordered list of data loaders, and run them (also partially)
forward and in reverse. The data loaders can be written in SQL or Python. You
must keep in mind the SQL loader is executed in the destination DB, so can be
useful to transform data between two schemas of same DB, and do it fast. The
Python loader selects the data and INSERTS/UPDATES it to the destination, so can
operate on different source and destination DB.(TODO: We should explore usingSELECTto CSV file and bulk insert into
destination DB).Keanu CLIKeanu command lives inclidirectory. It is a multitool consisting of following subcommands:schema- create tables from keanu schema in a database, or drop tables from a database.load- runs loaders to load data into destination db from source(s). It can run in full or incremental mode. Can run loaders all at once or in steps.delete- deletes data loaded by particular loader. Lets you "rewind" the work done by a particular loader (only full mode). This way you can "step back" in loading data.SetupInstall Pipenv command in your system:sudo apt install pipenvInstall dependencies inclidirectory:pipenv installConfigurationDepending on subcommand, you configure keanu cli with:environment variables (mostly for credentials eg API keys, or access urls, eg. url to database with user, pass, hostname, db name)configuration YAML file (when more structured configuration is needed eg. to specify sources and destination for loaders, as well as which loaders are used)command line switches and argumentsEnvironment variables referenceAPI keys and Database access urlsKeanu uses Database credentials in form of database url in format suitable for SQLalchemy. Because keanu can have many sources and another destination, the urls are provided using a yaml configuration.Configuration fileThe configuration file must have.yamlextension and contain alist of elements that define sources of data, destination of data, and transforms, which run a series of loaders.
You provide the configuration file as a positional argument toloadordeletesubcommands.SourceThe source should have a name, and db specification can either be url or schema. In case that source is the same db as destination, we say it islocalto that destination, and we can provide just schema name usingschemakey. You can use${VARIABLE}syntax to insert env variables (so, for instance, you do not store database credentials in configuration file).Example: source with database url, namedfoobarcrm- source:
    name: foobarcrm
    db:
      url: ${DATABASE_URL}Example: source in the same DB as destination, but inexchange_ratesdatabase- source:
    name: currency_data
    db:
      schema: exchange_ratesDestinationThere can be only one destination in the config file (so there is no need to give it a name). The SQL transforms are run
In the destination DB (they are SQL statements run on the DB server). This means
sources must be local to the destination for SQL loaders. Keanu cannot yet
stream data from one DB and insert it to another DB, but this is a feature we
would like to develop in the feature (in this case the transform should run in
the source and return a dataset that can be directly inserted to the
destination).The python loaders run inside the keanu cli process, and download/upload data. They can use different db servers for sources and destination.Example:- destination:
    timezone: Europe/Berlin
    region: eu-west-1 
    db:
      url: ${DATABASE_URL}You can specify additional keys undersourceas well as underdestination. The whole dictionary is available underconfigproperty, eg.batch.destination.config. Some of them get special treatment:timezonekey is parsed by Python and available to python loaders usingbatch.destination.timezoneproperty.TransformThe transform points to directory with loaders and specifies their type. Transform is applied to one source and one destination. Because there can be many sources, transform should specify name of the sources undersourcekey. Transform can point todirectoryor just onefile.Example: transform will execute all SQL files found undersome/directory. It will usefoobarcrmas source.- transform:
    source: foobarcrm
    sql:
      directory: some/directoryExample: a transform using just one python loader file.- transform:
    source: foobarcrm
    py:
      file: python_scripts/conversions.pyYou can  Copy thecli/.env.samplefile to.envand change it to taste.Default settingHowever, keanu can also run with minamal default configuration.
Runningkeanu load sql_filesis equal tokeanu load config1.yamlwithconfig1.yaml:- source:
    db:
      schema: ${SOURCE}
- destination:
    db:
      url: ${DATABASE_URL}
- transform:
    sql:
       directory: ${ARGV[1]}DATABASE_URL- must use schema understood by SQLAlchemy library, required for load and delete subcommandsSOURCE- is a name of CiviCRM database (also called schema), required only for SQL loadersUsageKeanu commands has a set of subcommands, similar to git or heroku.Running from sourceInstall dependencies usingpipenv insalland open shell with needed requirements withpipenv shell.Package buildingTo build the package usepython setup.py bdist_wheel.To upload it to PyPI:Create a~/.pypircfile:[distutils] 
index-servers=pypi
[pypi] 
repository = https://upload.pypi.org/legacy/ 
username = pypi-usernameTo install the package locally:pip install --upgrade dist/keanu-0.1-py3-none-any.whlUsepython -m twine upload dist/*.SQL commandsThese are the commands related to creating and filling in the Keanu schema (from CiviCRM DB).schema: manipulating database schemakeanu.py schema -L FILENAME- loads SQL schema file.keanu.py schema -D- drops all tables from the database.load: loading datakeanu.py load CONFIGFILEloads data from source to destination db in ordered steps. The queries to run are specified in the configuration file.
Currently we support two kinds of loaders:SQL loader - consists of SQL file, containing special metadata in comments. It is run in the destination database server, and the source data must be available on the same server (in another database/schema).Python loader - consists of Python file, and is run on the local machine. Keanu bundles sqlalchemy and pandas libraries in case you need them.Loaders are run in order specified in each loader's metadata (ORDERkey), unless they are ingored (IGNOREkey). See other metadata keys in sql and py loader reference.Options:-o 10:-order, start from SQL file with order 10 (also see order spec)-n-dry run, do not execute the SQL. Usefull to see the order of scripts-i-incremental, also run the incremental parts of the SQL, normally deleted (they are marked by-- BEGIN INCREMENTALand-- END INCREMENTALcomments)-d-displaya whole SQL statement instead of abbreviation of first linedelete: deleting datakeanu.py delete SQLDIRwill go through the SQL scripts inreverse orderand execute the commentedDELETEandTRUNCATEstatements in the scripts.Options:-n-dry run, do not execute the SQL. Usefull to see the order of scripts-o 10:- delete data until the SQL fileorder(inclusive) and stop-d-displaya whole SQL statement instead of abbreviation of first lineOrder specYou can provide just a number to order (-o) option, but you can also provide:a list of numbers such as10,11,50and scripts with respective order will be runa range in python format such as10:16equal to10,11,12,13,14,15(excluding 16)a mix of them:10:13,31,50:52SQL file reference:We add metadata to SQL files in SQL comments. This is usefull because you can still work with the SQL files using tools like MySQL Workbench, or mysql command - the commands will be ignored.Supported commands are:-- ORDER: arg- arg should be a 3 digit number. It specifies the order this script will be run.-- DELETE FROM ..- a DELETE statement will be saved to revert this script, so to remove all the data it has added. Must start with all caps DELETE.-- TRUNCATE- similar to above. Please note: TRUNCATE will fail if any other table has reference to this table, even when it's empty.-- IGNORE- the rest of the file will be ignored.Blocks are also supported. They are marked by a pair of lines:-- BEGIN keyword..someSQLcodehere..-- END keywordThey should be properly nested.Supported blocks are:BEGIN INCREMENTAL/END INCREMENTAL- marks part of the SQL that will normally be removed, and will be only used if-i(incremental flag) is used while loading. You can have many INCREMENTAL blocks. Example use:-- this scripts inserts into cookie jar-- ORDER: 10-- DELETE FROM jarINSERTINTOjar(colour,size,external_id)SELECTcd.name,c.size,c.idFROMcookiecJOINcolour_dictionarycdONc.colour_id=cd.id-- when we do incremental, we exclude external_id's we already have:-- BEGIN INCREMENTALWHEREc.idNOTIN(SELECTexternal_idFROMjar)-- END INCREMENTALBEGIN INITIAL/END INITIAL- marks part of the SQL that will be removed in incremental-imode.Enviroment variable interpolation. Similar to shell, simple variable interpolation is supported.${FOO}will be interpolated into value of FOO environment variable. This is currently used to insert the source schema name with${SOURCE}.Working with TimezonesPython has a myriad ways to work with timezones, which can be confusing.Remember:When you run a Python loader, you need to manage the timezone of all datetimes.When you run an SQL loader, it is run inside the DB server, so the connection/server timezone appliesWe recommend:frompytzimporttimezone- to create a timezone, call it with TZ name:timezone('Europe/Warsaw')ortimezone('CET'). There is alsotimezone.utcshorthand.fromdatetimeimportdatetime- datetimes can havetzinfoset, and
always make sure it is in your app. Then you can be sure that TZ is tracked no
matter what, and you only need to make sure to be careful about input and
output of such datetimes.always pass tz to datetime constructors, for example:datetime.now(tz=abc);
otherwise you get a naive datetime with no timezone info (but in local time of
the machine).when reading datetime from string withdatetime.fromisoformatalso use
strings with timezone infodatetime.fromisoformat("2021-01-02T10:10:10+01:00").Mysql version below 8 do not properly handle timezones (they have not datetime
with timezone) - so if your mysql/mariadb server is in a non-UTC timezone, you
will need to convert the datetime to server timezone
(dt.astimezone(server_tz)) and then store it (with SQLAlchemty for example).
The TZ will just be dropped, but the result is what you want. (You can
consider migrating your server to use UTC to make this all easier).ContributingTODOCode Linting and FormattingA pylint configuration is in pylintrc - configure your editor to run it automatically. Use black for formatting.