Package: elt-tools-aio
ELT-Tools-AIODatabase AbstractionA set of tools to serve as an abstraction layer over many commonly used databases, as long
as it's supported by SQLAlchemy. It supports the following operations in an easy-to-use
interface:asynchronous (non-blocking) operationcount the number of rows in a tablefind duplicates in a tablefind records missing in target with respect to sourcefind records on target which have been hard deleted from sourceexecute a sql query against a tableELT Pair OperationsIn Extract-Load-Transform (ELT) operations, a table is extracted and loaded from one database
to another with potential transformations after that (for example in a database view). This is
akin to database replication, albeit not necessarily all tables nor all columns are transferred.
One may also only transfer records from a certain date onwards.Many common database engineering tasks relate to the source and target pairs. This library
assists by implementing these commonly performed operations in a succinct interface such as:show a list of common tables between source and target databasecompare counts between source and target tables over a specified time windowfind primary keys of missing records in the targetfill missing records into the target over a given date rangefind primary keys of orphaned records in the target (i.e. corresponding records from the
source database have been deleted)remove orphaned records from target (even for large tables)Configuration and ExamplesThe library provides two main classes:DataClientfor database abstraction andELTDBPairfor
ELT operations between database pairs. The user passes configuration dictionaries into these classes.
The configuration describes database credentials, and details of which databases to pair up.For example, to find duplicate on a particular table:importasynciofromosimportenvironfromelt_tools_aio.clientimportDataClientFactoryDATABASES={'db_key11':{'engine':'oltp_engine','sql_alchemy_conn_string':environ.get('mysql_db_uri'),},'db_key12':{'engine':'bigquery_engine','dataset_id':'mydata','gcp_project':environ.get('GCP_PROJECT'),'gcp_credentials':environ.get('GOOGLE_APPLICATION_CREDENTIALS'),},}asyncdefprint_duplicate_keys():factory=DataClientFactory(DATABASES)client=factory(db_key='db_key11')customer_duplicates=awaitclient.find_duplicate_keys('customers','id')print(customer_duplicates)asyncio.run(print_duplicate_keys())For example, to remove orphaned records on the target table of a particular ELT Pair
using a binary search strategy on a large table:importasynciofromosimportenvironfromelt_tools_aio.clientimportELTDBPairFactoryDATABASES={'db_key11':{'engine':'oltp_engine','sql_alchemy_conn_string':environ.get('mysql_db_uri'),},'db_key12':{'engine':'bigquery_engine','dataset_id':'mydata','gcp_project':environ.get('GCP_PROJECT'),'gcp_credentials':environ.get('GOOGLE_APPLICATION_CREDENTIALS'),},}ELT_PAIRS={'pair1':{'source':'db_key11','target':'db_key12'},}asyncdefremove_orphans():factory=ELTDBPairFactory(ELT_PAIRS,DATABASES)elt_pair=factory(pair_key='pair1')_=awaitelt_pair.remove_orphans_from_target_with_binary_search('customers','id',timestamp_fields=['created_at'])asyncio.run(remove_orphans())Installation instructions$pipinstallgit+ssh://git@github.com/dewaldabrie/elt_tools_aio.git