Package: dwopt
Getting summary statistics out of database tables is often an unstreamlined process.
Does one read in millions of rows before doing any work on Python,
or run SQL elsewhere and use intermediate CSVs,
or write sql strings in python scripts?The Python packagedwopt(Datawarehouse Operator)
uses classes with a collection of sql templates to dynamically build and run queries,
under a flexible summary query building framework.Specifically, it features Excel-pivot-table-like API,
a collection of dataframe-summary-methods-like API,
and a collection of DDL/DML statements, metadata query wrappers.Supports:Python 3.10, 3.11.Windows 10: Sqlite, Postgres, Oracle.Linux: Sqlite, Postgres.See the Features and the Walk Through section for examples.Installationpip install dwoptInstall the database drivers for the database engines you want to use.pip install psycopg2 # postgres
pip install psycopg2-binary # in case can't build psycopg2

pip install oracledb # oracleFeaturesRun sql frictionlessly using saved credentialsRun sql script with text replacementProgramatically make simple sql queryTemplates: Excel-pivot-table-like APITemplates: Dataframe-summary-methods-like APITemplates: DDL/DML statements, metadata queriesStandard logging with reproducible sqlWalk ThroughRun sql frictionlessly using saved credentialsOn import, the package gives ready-to-be-useddatabase operator objectswith default credentials, which could be saved prior by user to
the system keyring using thedwopt.save_urlfunction.fromdwoptimportltlt.iris()lt.run('select count(1) from iris')count0150This enable quick analysis from any Python/Console window:fromdwoptimportpgpg.iris()pg.qry('iris').valc('species','avg(petal_length)')speciesnavg(petal_length)0sicolor504.2601setosa501.4622rginica505.552Alternatively, use the database operator object factory functiondwopt.dband the database engine url to access database.fromdwoptimportdbd=db("postgresql://dwopt_tester:1234@localhost/dwopt_test")d.mtcars()d.run('select count(1) n from mtcars')n032Allows for thick mode connection to Oracle database:fromdwoptimportdburl="""oracle+oracledb://dwopt_test:1234@localhost:1521/?service_name=XEPDB1
&encoding=UTF-8&nencoding=UTF-8"""lib_dir="C:/app/{user_name}/product/21c/dbhomeXE/bin"o=db(url,thick_mode={"lib_dir":lib_dir})o.run("select * from dual")dummy0XSeeTestingsection for package version tested.Run sql script with text replacementUse the database operator object’srunmethod to run sql script file.
One could then replace:marked parameters via mappings supplied to the method.Colon syntax is to be deprecated.
A future version will use jinja2 syntax across the board.fromdwoptimportpg,make_test_tbl_=make_test_tbl(pg)pg.run(pth="E:/projects/my_sql_script.sql",my_run_dte='2022-03-03',my_label='20220303',threshold=5)count0137Above runs the sql stored onE:/projects/my_sql_script.sqlas below:droptableifexistsmonthly_extract_:my_label;createtablemonthly_extract_:my_labelasselect*fromtestwheredte=to_date(':my_run_dte','YYYY-MM-DD')andscore>:threshold;selectcount(1)frommonthly_extract_:my_label;Programatically make simple sql queryThe database operator object’sqrymethod returns thequery object.
Use it’slist of clause methodsto make a simple sql query.This is not faster than just writing the sql,
main usage is to provide flexibility to thesummary query building framework.fromdwoptimportltlt.mtcars()sql="select cyl from mtcars group by cyl having count(1) > 10"q=(lt.qry('mtcars a').select('a.cyl, count(1) n, avg(a.mpg)').case('cat',"a.cyl = 8 then 1",els=0).join(f'({sql}) b','a.cyl = b.cyl',how='inner').group_by('a.cyl').having('count(1) > 10').order_by('n desc'))q.run()cylnavg(a.mpg)cat081415.1000001141126.6636360q.print()selecta.cyl,count(1)n,avg(a.mpg),casewhena.cyl=8then1else0endascatfrommtcarsainnerjoin(selectcylfrommtcarsgroupbycylhavingcount(1)>10)bona.cyl=b.cylgroupbya.cylhavingcount(1)>10orderbyndescTemplates: Excel-pivot-table-like APIUse thequery objectand it’svalcmethod to make and run
a value counts summary query with custom groups and calcs,
on top of arbituary sub-query, as part of thesummary query building framework.Then call the result dataframe’spivotmethod to finalize the pivot table.fromdwoptimportlt,make_test_tbl_=make_test_tbl(lt)(lt.qry('test').where('score>0.5','dte is not null','cat is not null').valc('dte,cat','avg(score) avgscore, round(sum(amt)/1e3,2) total').pivot('dte','cat'))Result:catnavgscoretotaldtetesttraintesttraintesttrain2022-01-01114010512.7362752.800106565.67530.092022-02-02107711002.7590612.748898536.68544.102022-03-03103710722.7285272.743825521.54528.85The final query used can be invoked by thevalcmethod, or logged via standard
logging.withxas(select*fromtestwherescore>0.5anddteisnotnullandcatisnotnull)selectdte,cat,count(1)n,avg(score)avgscore,round(sum(amt)/1e3,2)totalfromxgroupbydte,catorderbyndescTemplates: Dataframe-summary-methods-like APIUse thequery objectand it’slist of summary methodsto make and run
summary queries on top of arbituary sub-query,
as part of thesummary query building framework:fromdwoptimportpgpg.iris()q=pg.qry('iris a').select('a.*').case('cat',"petal_length > 5             then '5+'","petal_length between 2 and 5 then '2-5'","petal_length < 2             then '-2'",)#Column names:q.cols()['sepal_length','sepal_width','petal_length','petal_width','species','cat']#Number of distinct combination:q.dist(['species','petal_length'])count48Name:0,dtype:int64#Head:q.head()sepal_lengthsepal_widthpetal_lengthpetal_widthspeciescat05.13.51.40.2setosa-214.93.01.40.2setosa-224.73.21.30.2setosa-234.63.11.50.2setosa-245.03.61.40.2setosa-2#Length:q.len()150#Min and max value:q.mimx('petal_length')max6.9min1.0Name:0,dtype:float64#Top record:q.top()sepal_length5.1sepal_width3.5petal_length1.4petal_width0.2speciessetosacat-2Name:0,dtype:object#Value count followed by pivot:q.valc('species, cat').pivot('species','cat','n')cat-22-55+speciesrginicaNaN9.041.0setosa50.0NaNNaNsicolorNaN49.01.0#--All summary methods support output by printing or str:q.valc('species, cat',out=1)withxas(selecta.*,casewhenpetal_length>5then'5+'whenpetal_lengthbetween2and5then'2-5'whenpetal_length<2then'-2'elseNULLendascatfromirisa)selectspecies,cat,count(1)nfromxgroupbyspecies,catorderbyndescTemplates: DDL/DML statements, metadata queriesUse thelist of operation methodsto make and run some
DDL/DML statements with convenient or enhanced functionalities:importpandasaspdfromdwoptimportlttbl=pd.DataFrame({'col1':[1,2],'col2':['a','b']})tbl2=pd.DataFrame({'col1':[1,3],'col2':['a','c']})lt.drop('test')lt.create('test',col1='int',col2='text')lt.write(tbl,'test')lt.write_nodup(tbl2,'test',['col1'],"col1 < 4")lt.run("select * from test")col1col201a12b23clt.drop('test')lt.cwrite(tbl,'test')lt.qry('test').run()col1col201a12bUse thelist of metadata methodsto make and run some useful metadata queries:fromdwoptimportpgpg.iris()pg.table_cols('public.iris')column_namedata_type0sepal_lengthreal1sepal_widthreal2petal_lengthreal3petal_widthreal4speciescharactervaryingfromdwoptimportltlt.iris()lt.mtcars()lt.list_tables().iloc[:,:-1]typenametbl_namerootpage0tableirisiris21tablemtcarsmtcars5Standard logging with reproducible sqlMany of the package’s methods are wired through the standardloggingpackage. In particular, therunmethod emits sql used asINFOlevel message.
The relevant logger object has standard naming and is calleddwopt.dbo.Example configuration to show logs in console:importlogginglogging.basicConfig(level=logging.INFO)fromdwoptimportltlt.iris(q=1).valc('species','avg(petal_length)')INFO:dwopt.dbo:dropping table via sqlalchemy: iris
INFO:dwopt.dbo:done
INFO:dwopt.dbo:creating table via sqlalchemy:
INFO:dwopt.dbo:('sepal_length', Column('sepal_length', REAL(), table=<iris>))
INFO:dwopt.dbo:('sepal_width', Column('sepal_width', REAL(), table=<iris>))
INFO:dwopt.dbo:('petal_length', Column('petal_length', REAL(), table=<iris>))
INFO:dwopt.dbo:('petal_width', Column('petal_width', REAL(), table=<iris>))
INFO:dwopt.dbo:('species', Column('species', String(), table=<iris>))
INFO:dwopt.dbo:done
INFO:dwopt.dbo:running:
INSERT INTO iris (sepal_length, sepal_width, petal_length, petal_width, species) VALUES (:sepal_length, :sepal_width, :petal_length, :petal_width, :species)
INFO:dwopt.dbo:args len=150, e.g.
{'sepal_length': 5.1, 'sepal_width': 3.5, 'petal_length': 1.4, 'petal_width': 0.2, 'species': 'setosa'}
INFO:dwopt.dbo:done
INFO:dwopt.dbo:running:
with x as (
    select * from iris
)
select
    species
    ,count(1) n
    ,avg(petal_length)
from x
group by species
order by n desc
INFO:dwopt.dbo:done
   species   n  avg(petal_length)
0  sicolor  50              4.260
1   setosa  50              1.462
2  rginica  50              5.552Alternatively, to avoid logging info messages from other packages:importlogginglogging.basicConfig()logging.getLogger('dwopt.dbo').setLevel(logging.INFO)Example configuration to print on console and store on file with timestamps:importlogginglogging.basicConfig(format="%(asctime)s[%(levelname)s]%(message)s",handlers=[logging.FileHandler("E:/projects/logs.log"),logging.StreamHandler()])logging.getLogger('dwopt.dbo').setLevel(logging.INFO)Debug logging:importlogginglogging.basicConfig()logging.getLogger('dwopt').setLevel(logging.DEBUG)Sqlalchemy logger can also be used to obtain even more details:importlogginglogging.basicConfig()logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)DevelopmentInstallationTesting, documentation building package:#venvonlinuxsudo apt-get install python3-venv
python3.11 -m venv dwopt_dev
source dwopt_dev/bin/activate
deactivate#testingpython -m pip install pytest black flake8 tox#docoandpackagingpython -m pip install sphinx sphinx_rtd_theme build twine wheel#dependpython -m pip install -U sqlalchemy pandas keyring
python -m pip install -U keyrings.alt
python -m pip install -U psycopg2
python -m pip install -U oracledb#considerpython -m pip install -U psycopg2-binary
python -m pip install -U cx_Oracle#packagepython -m pip install -e .TestingTest:python -m toxTesting for specific databases.
Set up environment based ondwopt.make_test_tblfunction notes.python -m pytest
python -m pytest --db=pg
python -m pytest --db=ocTest code styles:flake8 src/dwoptDatabases used for testings are:Postgres 15
Oracle express 21cPackage versions tested are:Name: keyring
Version: 24.2.0
---
Name: keyrings.alt
Version: 5.0.0
---
Name: oracledb
Version: 1.3.2
---
Name: pandas
Version: 2.0.3
---
Name: psycopg2-binary
Version: 2.9.6
---
Name: SQLAlchemy
Version: 2.0.19DocumentationBuild document:cd docs
make htmlTest examples across docs:cd docs
make doctestDocumentationAPI