Package: db-analytics-tools
DB Analytics ToolsDatabases Analytics Tools is a Python open source micro framework for data analytics. DB Analytics Tools is built on top of Psycopg2, Pyodbc, Pandas, Matplotlib and Scikit-learn. It helps data analysts to interact with data warehouses as traditional databases clients.Why adopt DB Analytics Tools ?Easy to learn : It is high level API and doesn't require any special effort to learn.Real problems solver : It is designed to solve real life problems of the Data AnalystAll in One : Support queries, Data Integration, Analysis, Visualization and Machine LearningCore Components#ComponentDescriptionHow to import0dbDatabase Interactions (Client)import db_analytics_tools as db1dbiData Integration & Data Engineeringimport db_analytics_tools.integration as dbi2dbaData Analysisimport db_analytics_tools.analytics as dba3dbvizData Visualizationimport db_analytics_tools.plotting as dbviz4dbmlMachine Learning & MLOpsimport db_analytics_tools.learning as dbmlInstall DB Analytics ToolsDependenciesDB Analytics Tools requiresPythonPsycopg2PyodbcPandasSQLAlchemyStreamlitDB Analytics Tools can easily installed using pippipinstalldb-analytics-toolsGet StartedSetup clientAs traditional databases clients, we need to provide database server ip address and port and credentials. DB Analytics Tools supports Postgres and SQL Server.# Import DB Analytics Toolsimportdb_analytics_toolsasdb# Database Infos & CredentialsENGINE="postgres"HOST="localhost"PORT="5432"DATABASE="postgres"USER="postgres"PASSWORD="admin"# Setup clientclient=db.Client(host=HOST,port=PORT,database=DATABASE,username=USER,password=PASSWORD,engine=ENGINE)Data Definition Languagequery="""----- CREATE TABLE -----drop table if exists public.transactions;create table public.transactions (transaction_id integer primary key,client_id integer,product_name varchar(255),product_category varchar(255),quantity integer,unitary_price numeric,amount numeric);"""client.execute(query=query)Data Manipulation Languagequery="""----- POPULATE TABLE -----insert into public.transactions (transaction_id, client_id, product_name, product_category, quantity, unitary_price, amount)values(1,101,'Product A','Category 1',5,100,500),(2,102,'Product B','Category 2',3,50,150),(3,103,'Product C','Category 1',2,200,400),(4,102,'Product A','Category 1',7,100,700),(5,105,'Product B','Category 2',4,50,200),(6,101,'Product C','Category 1',1,200,200),(7,104,'Product A','Category 1',6,100,600),(8,103,'Product B','Category 2',2,50,100),(9,103,'Product C','Category 1',8,200,1600),(10,105,'Product A','Category 1',3,100,300);"""client.execute(query=query)Data Query Languagequery="""----- GET DATA -----select *from public.transactionsorder by transaction_id;"""dataframe=client.read_sql(query=query)print(dataframe.head())transaction_id  client_id product_name product_category  quantity  unitary_price  amount
0               1        101    Product A       Category 1         5          100.0   500.0
1               2        102    Product B       Category 2         3           50.0   150.0
2               3        103    Product C       Category 1         2          200.0   400.0
3               4        102    Product A       Category 1         7          100.0   700.0
4               5        105    Product B       Category 2         4           50.0   200.0Implement SQL based ETLETL API is in the integration moduledb_analytics_tools.integration. Let's import it ans create an ETL object.# Import Integration moduleimportdb_analytics_tools.integrationasdbi# Setup ETLetl=dbi.ETL(client=client)ETLs for DB Analytics Tools consists in functions with date parameters. Everything is done in one place i.e on the database. So first create a function on the database like this :query="""----- CREATE FUNCTION ON DB -----create or replace function public.fn_test(rundt date) returns integerlanguage plpgsqlas$$begin--- DEBUG MESSAGE ---raise notice 'rundt : %', rundt;--- EXTRACT ------ TRANSFORM ------ LOAD ---return 0;end;$$;"""client.execute(query=query)Run a functionThen ETL function can easily be run using the ETL class via the methodETL.run()# ETL FunctionFUNCTION="public.fn_test"## Dates to runSTART="2023-08-01"STOP="2023-08-05"# Run ETLetl.run(function=FUNCTION,start_date=START,stop_date=STOP,freq="d",reverse=False)Function    : public.fn_test
Date Range  : From 2023-08-01 to 2023-08-05
Iterations  : 5
[Runing Date: 2023-08-01] [Function: public.fn_test] Execution time: 0:00:00.122600
[Runing Date: 2023-08-02] [Function: public.fn_test] Execution time: 0:00:00.049324
[Runing Date: 2023-08-03] [Function: public.fn_test] Execution time: 0:00:00.049409
[Runing Date: 2023-08-04] [Function: public.fn_test] Execution time: 0:00:00.050019
[Runing Date: 2023-08-05] [Function: public.fn_test] Execution time: 0:00:00.108267Run several functionsMost of time, several ETL must be run and DB Analytics Tools supports running functions as pipelines.## ETL FunctionsFUNCTIONS=["public.fn_test","public.fn_test_long","public.fn_test_very_long"]## Dates to runSTART="2023-08-01"STOP="2023-08-05"# Run ETLsetl.run_multiple(functions=FUNCTIONS,start_date=START,stop_date=STOP,freq="d",reverse=False)Functions   : ['public.fn_test', 'public.fn_test_long', 'public.fn_test_very_long']
Date Range  : From 2023-08-01 to 2023-08-05
Iterations  : 5
*********************************************************************************************
[Runing Date: 2023-08-01] [Function: public.fn_test..........] Execution time: 0:00:00.110408
[Runing Date: 2023-08-01] [Function: public.fn_test_long.....] Execution time: 0:00:00.112078
[Runing Date: 2023-08-01] [Function: public.fn_test_very_long] Execution time: 0:00:00.092423
*********************************************************************************************
[Runing Date: 2023-08-02] [Function: public.fn_test..........] Execution time: 0:00:00.111153
[Runing Date: 2023-08-02] [Function: public.fn_test_long.....] Execution time: 0:00:00.111395
[Runing Date: 2023-08-02] [Function: public.fn_test_very_long] Execution time: 0:00:00.110814
*********************************************************************************************
[Runing Date: 2023-08-03] [Function: public.fn_test..........] Execution time: 0:00:00.111044
[Runing Date: 2023-08-03] [Function: public.fn_test_long.....] Execution time: 0:00:00.123229
[Runing Date: 2023-08-03] [Function: public.fn_test_very_long] Execution time: 0:00:00.078432
*********************************************************************************************
[Runing Date: 2023-08-04] [Function: public.fn_test..........] Execution time: 0:00:00.127839
[Runing Date: 2023-08-04] [Function: public.fn_test_long.....] Execution time: 0:00:00.111339
[Runing Date: 2023-08-04] [Function: public.fn_test_very_long] Execution time: 0:00:00.140669
*********************************************************************************************
[Runing Date: 2023-08-05] [Function: public.fn_test..........] Execution time: 0:00:00.138380
[Runing Date: 2023-08-05] [Function: public.fn_test_long.....] Execution time: 0:00:00.111157
[Runing Date: 2023-08-05] [Function: public.fn_test_very_long] Execution time: 0:00:00.077731
*********************************************************************************************DocumentationDocumentation available onhttps://joekakone.github.io/db-analytics-tools.Help and SupportIf you need help on DB Analytics Tools, please send me an message onWhatsappor send me amail.ContributingPlease see the contributing docs.MaintainerDB Analytics Tools is maintained byJoseph Konka. Joseph is a Data Science Professional with a focus on Python based tools. He developed the base code while working at Togocom to automate his daily tasks. He packages the code into a Python package calledSQL ETL Runnerwhich becomesDatabases Analytics Tools. For more about Joseph Konka, please visitwww.josephkonka.com.Let's get in touch