Package: dbqq
Database Quick QueryDatabase Quick QueryEnvironment VariablesInitializationConnection ConfigurationParameter StoresEncrypting ConfigsCreating Private and Public KeysEncrypting The Configuration FileConnectorsCachingGetting Basic Table MetadataRunning Queries from a FileExtracting queries and connections from a fileBasic Connection No CacheConnection With CacheConnection With Cache and Date Lower BoundParsing FilesJinja TemplatesCommon Table ExpressionsRollbackDatabricks DevelopmentCLI Toolsdbqq-clean-connectionsdbqq-run-sqlA wrapper over various database connector libraries for quickly performing
queries for analysis. The package leveragespolarsandconnectorxfor their speed when handling Big data.Environment VariablesExample environment variable setup# path to private key$env:DBQQ_PRIVATE_KEY=".\private_key.pem"or".\private_key.der"or"none"# path to connections, supports encrypted or flat yaml files$env:DBQQ_CONNECTORS=".\connections.dbqq"or".\connections.yaml"# if we have aws parameter store configuration$env:DBQQ_SSM_NAME="name of parameter"InitializationConnections are loaded dynamically, and are initially blank within the
connection files# dbqq/connectors/databricks.py etc.#! begin inject regex#! end inject regexTo initialize connections, simply install the package and run the commanddbqq-initialize-connectorsAs an example, after importing the package the configured connections should now
be visible within the connection files. The following will be the result of an
initialization given the configs in theConnection Configurationsection.# dbqq/connectors/databricks.py#! begin inject regexclasspreprod(_general_connector):source:str='preprod'classprod(_general_connector):source:str='prod'classtest(_general_connector):source:str='test'classdev(_general_connector):source:str='dev'#! end inject regexConnection ConfigurationTo configure login details for various databases fill out the following required
details into a.yaml(optionally now encrypt the file and delete the yaml)
and set the environment variableDBQQ_PRIVATE_KEYto the location of the file.databricks:dev:access_token:"*******"http_path:"*******"server_hostname:"*******"test:access_token:"*******"http_path:"*******"server_hostname:"*******"preprod:access_token:"*******"http_path:"*******"server_hostname:"*******"prod:access_token:"*******"http_path:"*******"server_hostname:"*******"mssql:conn1:authentication_type:"*******"database:"*******"driver:"*******"hostname:"*******"password:"*******"port:"*******"username:"*******"conn2:authentication_type:"*******"database:"*******"driver:"*******"hostname:"*******"password:"*******"port:"*******"username:"*******"oracle:conn1:authentication_type:"*******"connection_type:"*******"database:"*******"hostname:"*******"password:"*******"port:"*******"role:"*******"username:"*******"conn2:authentication_type:"*******"connection_type:"*******"database:"*******"hostname:"*******"password:"*******"port:"*******"role:"*******"username:"*******"Parameter StoresParameters can be extracted from the AWS Parameter Store, which overrides local
configs. To retrieve the parameters, set thenameand theregion. By default,
the region is set to `ap-southeast-2``.$env:DBQQ_SSM_NAME="name"$env:DBQQ_SSM_REGION="region"Encrypting ConfigsIt is probably not the best idea to store passwords as a flat file on your local
computer. We have added the ability to encrypt configuration files via thedbqq-encrypt-yaml.exe.Creating Private and Public KeysBefore we can encrypt our configuration file we need to create both private and
public keys. For our example let's say we have a working directory
with the following configuration fileC:\devops>lsModeLastWriteTimeLengthName----------------------------a----config.yamlcreate a folder called keysC:\devops>mkdirkeyswe can use thedbqq-write-keys.exemethod to generate the private and public
keysC:\devops>dbqq-write-keys-l'./keys'-k1024# supports der or pem call -h for helpC:\devops>lskeysModeLastWriteTimeLengthName----------------------------a----private_key.der-a----public_key.derEncrypting The Configuration FileApplydbqq-encrypt-yamlon the config file with the generated private keyC:\devops>dbqq-encrypt-yaml'config.yaml''./keys/private_key.der'C:\devops>lskeysModeLastWriteTimeLengthName----------------------------a----config.dbqqFinally secure the private keyConnectorsThe connectors module reads from the configuration files, categorizing the
details into (currently) either anoracle,mssqlordatabricksconnection.Given our example configuration file we can create anoracleto dbconn1by
calling:importpolarsasplfromdbqqimportconnectorsconnection=connectors.oracle.db1()to run a query simply call the connectiondf:pl.LazyFrame=connection("select * from some.table")connection calls always return a polarsLazyFrame.
This is because LazyFrames allow for whole-query optimisation in addition to
parallelism, making them the preferred (and highest-performance) mode of
operation for polars.Runcollectto work with a DataFrame orfetchto partially collectdf:pl.Dataframe=df.collect()CachingDuring the analysis, we may want to cache queries for iteration. This can be
performed simply by calling the cache method:connection.cache()("select * from schema.table")the data will be saved as a.parquetfile with a corresponding.yamlfile.
The cached files will be stored in the.tempfolder with a unique identifier
unless specified.To set the location and name of the cached filesconnection.cache(directory="./cache_directory",name="name")("select * from schema.table")Finally, we might want to save a new cache every day while clearing out old
files. This can be done using thedate_lower_boundkeyword argument. Files
before this date will be deleted forallcached queries.fromdatetimeimportdatetime,timedeltaconnection.cache(date_lower_bound=datetime.now()-timedelta(days=1))("select * from schema.table")Getting Basic Table MetadataTo get information about the columns before running the query run thedescribe_columnsmethod on the table using the required connection.connection.describe("table_name")To get the schema of the table call theschemaattribute on the LazyFramedf:pl.LazyFrame=connection("select * from some.table")df.schemaRunning Queries from a FileTo run a query directly from a fileconnection.from_file("path to file",*args,**kwargs)where*args,**kwargsare the arguments for a regular__call__.Extracting queries and connections from a fileWe can include metadata to an sql file and call theconnectors.from_filemethod to extract both a the connection and query. There are 3 ways we can
include metadata.Basic Connection No CacheTo create a simple connection we can add the following configuration string
to the sql file.--! name/connector.{type}.{db}select*fromsome.tableReplace {} with connection information i.e. databricks.prod.The configuration string will be removed from the query.connection,query=connectors.from_file(filepath)df=connection(query)Connection With CacheTo create a connection with a path to a cache file add a string to the
configuration specifying the path to the cache folder. Thenameprovided will
be used as the name of the cache i.e.name.parquet. Ifname=_then a
unique identifier will be used as the name for the cache.--! name/"./temp"/connector.{type}.{db}select*fromsome.tableConnection With Cache and Date Lower BoundTo add a date lower bound add a command which can be evaluated by python. Note, datetim and timedelta
are imported during the eval process.--! name/"./temp"/datetime.now()-timedelta(days=7)/connector.{type}.{db}select*fromsome.tableParsing FilesWe might simply want to parse the files i.e. for automating tasks.fromdbqqimportutilsdata=utils.parse_file("<path to file>")data for parsed sql can be found in connectors.data.parsed.sqlJinja TemplatesWe can parse jinja templates, let's say we have a template calledtest_query.sql.j2select{%foritemincolumns%}{%ifloop.last%}{{item}}{%else%}{{item}},{%endif%}{%endfor%}fromsome.tableWe can create aRenderedTemplateLoaderfrom it using the followingimportdbqq.connectorsasdbcconn=dbc.<source>.<conn>()rendered_template_loader=conn.render_template("test_query.sql.j2",columns=["col1","col2","col3","col4"],)# will return a object that shows the renderedquery"""selectcol1,col2,col3,col4fromsome.table"""table=rendered_template_loader.execute()Common Table ExpressionsWe can construct a common table expression with the following methodfromdbqqimportutilsfromtriple_quote_cleanimportTripleQuoteCleanertqc=TripleQuoteCleaner(skip_top_lines=1)cte=utils.CommonTableExpression()cte.add_query("query_1","""--sqlselect *from table_1""">>tqc)cte.add_query("query_2","""--sqlselect*fromtable_2 t2inner join table_1 t1on t1.col1 = t2.col2""">>tqc)print(cte("select * from table_2"))outputwithquery_1as(select*fromtable_1),query_2as(select*fromtable_2t2innerjointable_1t1ont1.col1=t2.col2)select*fromtable_2RollbackWhen in a jupyter notebook we can rollback queries during the development processcte.rollback_one()# or rollback(version_no)cte.add_query(query_name,query)this allows us to modify the cte on the flyDatabricks Developmentsometimes we may want to develop databricks queries locally, and use spark when
in databricks.connection=dbqq.connectors.databricks.prod()...when in databricks the connection will be automatically converted into adbqq.connectors.databricks.Clusterobject, which now will use the spark context to run queries.CLI Toolsdbqq-clean-connectionsClean up all the connections, useful for a fresh startusage:dbqq-clean-connections.exe[-h]Cleanupallconnectioninformationoptionalarguments:-h,--helpshowthishelpmessageandexitdbqq-run-sqlrun the sql with the provided configurationsusage:dbqq-run-sql.exe[-h][-nROWS][-i{0,1}]fileRunaquerydirectlyfromanfilepositionalarguments:file*.sqlfiletoberunoptionalarguments:-h,--helpshowthishelpmessageandexit-nROWS,--rowsROWSthetotalnumberofrowstoshownumber,specify'all'toshowall-i{0,1},--show_index{0,1}showtheindexCombineparquet to csvto view/convert the output