Package: db-utils
db_utils libraryIntroductiondb-utils is a Python package that standardizes interactions with various types of databases.  db-utils is a collection of modules that lowers the bar to viewing, extracting, and analyzing data from various sources including:RedshiftSnowflakePostgresMysqlSqlserversqlites3dynamoDBDockerOne line command to start a Docker container with db-utils installed.  This will run on any system that is running Docker.  A jupyter notebook will open up on port 8888, just copy and paste the url from the terminal into your preferred browser.dockerrun-p8888:8888hannaj06/db-utils:latestdocker-composeBelow is an exampledocker-compose.yamlfile.  With this configuration jupyter notebook changes persist in the location defined by<local_notebooks>.  Below is a sample of the.databases.conffile which will be bindmounted to the docker image.docker-compose.yamlversion:"2.1"services:dbutils_juypter:image:hannaj06/db-utilsports:-8888:8888volumes:-${HOME}/.databases.conf:/root/.databases.conf-<local_notebooks>:/notebooks.databases.conf[redshift]host=<redshift_host>user=<user>password=<redshift_password>port=<port>database=<db>[s3]aws_access_key_id=<access_id>aws_secret_access_key=<secret_access>region=<aws_region>default_bucket=<default_bucket>Installation on local envoirnmentsudo apt-get updateRequired system packages:sudo apt-get install python3-dev (Ubuntu)sudo apt-get apt-get install g++ (Ubuntu)sudo apt-get install libpq-dev (Ubuntu)sudo apt-get install unixodbc-dev (Ubuntu)brew install postgresql (MacOS)pipinstalldb_utilspg_connect class (previously DBUtil)A database connection class to interact with  Postgres or RedshiftBasic Usage:create database configuration fileexample below is called .databases.conf[redshift_example]
    host=redshift.example.com
    user=test_user
    password=password
    port=5439
    database=test_db

    >>> from db_utils.pg_connect import pg_connect
    >>>
    >>> db = pg_connect('redshift_example', '.databases.conf')
    >>> db.get_arr_from_query('select * from test', pprint=True)snowflake_connect classA database connection class to interact with snowflakeBasic Usage:create database configuration fileexample below is called .databases.conf[snowflake]
    account=abc123.us-east-1
    host=abc123.us-east-1.snowflakecomputing.com
    user=test_user
    password=password
    port=443
    database=test_db
    aws_access_key_id=<key_id>
    aws_secret_access_key=<secret_key>snowflake_s3 classA child class of snowflake_connect class used to retrieve large datasets in small chunksBasic Usage:create database configuration fileexample below is called .databases.confnote the additional fields required[snowflake]
    account=abc123.us-east-1
    host=abc123.us-east-1.snowflakecomputing.com
    user=test_user
    password=password
    port=443
    database=test_db
    aws_access_key_id=<key_id>
    aws_secret_access_key=<secret_key>
    default_bucket=example) Loading large data set into memory in chunks>>> from db_utils.snowflake_connect import snowflake_s3
    >>> import os
    >>>
    >>> file_format = '''
    TYPE = CSV
    COMPRESSION = NONE
    '''
    >>>
    >>>
    >>> with snowflake_s3('snowflake', '.databases.conf') as db:
    >>>     db.cursor('SELECT * FROM example_large_table', file_format=file_format, pprint=True)
    >>>
    >>>     while True:
    >>>         file = db.fetch(contents=True)
    >>>
    >>>         if file:
    >>>             for row in file:
    >>>                 print(row)
    >>>
    >>>         else:
    >>>             breaksqlite_connect classA database connection class to interact with SQLite>>> from db_utils.sqlite_connect import sqlite_connect
    >>>
    >>> db = sqlite_connect('test.db')
    >>> db.get_df_from_query('select * from test_table', pprint=True)s3_connect classConnection library for interacting with S3Basic Usage:add s3 section to .databases.conf file (created in previous example)[s3]
        aws_access_key_id=<key_id>
        aws_secret_access_key=<secret_key>
        default_bucket=<bucket>


    >>> from db_utils.s3_connect import s3_connect
    >>>
    >>> s3 = s3_connect('.databases.conf', 's3')
    >>> s3.list_keys(prefix='examples')example) grab file from s3 into memory as stringIO object>>> from db_utils.s3_connect import s3_connect
    >>>
    >>> s3 = s3_connect('.databases.conf', 's3')
    >>> s3.get_contents('example_file', stringIO=True)
    >>> s3.read()sql_server connect classRequirements:sql server drivers -https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017Basic Usage:add sql server section to .databases.conf file (created in previous example)[sql_server]
    driver=ODBC Driver 17 for SQL Server
    server=127.0.0.1
    user=bill
    password=gates
    database=master

    >>> from db_utils.sql_server_connect import sql_server_connect
    >>> db = sql_server_connect('sql_server', 'databases.conf')
    >>>
    >>> db.get_arr_from_query('''SELECT * FROM SYSOBJECTS''', pprint=True))dynamodb_connect classConnection library for interacting with Dynamodbtimer classHelper class to time long running processesBasic Usage:>>> from db_utils.timer import timer
>>>
>>> t = timer()
>>> t.lap('s')
5.469961