Package: mara-db
Mara DBMini package for configuring and accessing multiple databases in a single project. Decouples the use of databases and their configuration by using "aliases" for databases.The filemara_db/dbs.pycontains abstract database configurations for PostgreSQL, Mysql, SQL Server, Oracle, SQLite and Big Query. The database connections of a project are configured by overwriting thedatabasesfunction inmara_db/config.py:importmara_db.configimportmara_db.dbs## configure database connections for different aliasesmara_db.config.databases=lambda:{'mara':mara_db.dbs.PostgreSQLDB(host='localhost',user='root',database='mara'),'dwh':mara_db.dbs.PostgreSQLDB(database='dwh'),'source-1':mara_db.dbs.MysqlDB(host='some-localhost',database='my_app',user='dwh'),'source-2':mara_db.dbs.SQLServerDB(user='dwh_read',password='123abc',database='db1',host='some-sql-server')}## access individual database configurations with `dbs.db`:print(mara_db.dbs.db('mara'))# -> <PostgreSQLDB: host=localhost, database=mara>Visualization of (PostgreSQL, MySQL, SQL Server) database schemasmara_db/views.pycontains a schema visualization for all configured databases using graphviz (currently PostgreSQL, Mysql and SQL Server only). It basically show tables of selected schemas together with the foreign key relations between them.For finding missing foreign key constraints, columns that follow a specific naming pattern (configurable viaconfig.schema_ui_foreign_key_column_regex, default*_fk) and that are not part of foreign key constraints are drawn in pink.Fast batch processing: Accessing databases with shell commandsThe filemara_db/shell.pycontains functions that create commands for accessing databases via their command line clients.For example, thequery_commandfunction creates a shell command that can receive an SQL query from stdin and execute it:importmara_db.shellprint(mara_db.shell.query_command('source-1'))# -> mysql --default-character-set=utf8mb4 --user=dwh --host=some-localhost my_appprint(mara_db.shell.query_command('dwh',timezone='Europe/Lisbon',echo_queries=False))# -> PGTZ=Europe/Lisbon PGOPTIONS=--client-min-messages=warning psql  --no-psqlrc --set ON_ERROR_STOP=on dwhThe functioncopy_to_stdout_commandcreates a shell command that receives a query on stdin and writes the result to stdout in tabular form:print(mara_db.shell.copy_to_stdout_command('source-1'))# -> mysql --default-character-set=utf8mb4 --user=dwh --host=some-localhost my_app --skip-column-namesSimilarly,copy_from_stdin_commandcreates a client command that receives tabular data from stdin and and writes it to a target table:print(mara_db.shell.copy_from_stdin_command('dwh',target_table='some_table',delimiter_char=';'))# -> PGTZ=Europe/Berlin PGOPTIONS=--client-min-messages=warning psql --echo-all --no-psqlrc --set ON_ERROR_STOP=on dwh \#      --command="COPY some_table FROM STDIN WITH DELIMITER AS ';'"Finally,copy_commandcreates a shell command that receives a sql query from stdin, executes the query insource_dband then writes the result of totarget_tableintarget_db:print(mara_db.shell.copy_command('source-2','dwh',target_table='some_table'))# -> sed 's/\\\\$/\$/g;s/\$/\\\\$/g' \#   | sqsh  -U dwh_read -P 123abc -S some-sql-server -D db1 -m csv \#   | PGTZ=Europe/Berlin PGOPTIONS=--client-min-messages=warning psql --echo-all --no-psqlrc --set ON_ERROR_STOP=on dwh \#         --command = "COPY some_table FROM STDIN WITH CSV HEADER"The followingcommand line clientsare used to access the various databases:DatabaseClient binaryCommentsPostgresql / RedshiftpsqlIncluded in standard distributions.MariaDB / MysqlmysqlIncluded in standard distributions.SQL Serversqsh- or -sqlcmdsqsh: Fromhttps://sourceforge.net/projects/sqsh/, usually messy to get working. On ubuntu, usehttp://ppa.launchpad.net/jasc/sqsh/ubuntu/backport. On Mac, try the homebrew version or install from source.sqlcmd: Official Microsoft Utility for SQL Server. Seesqlcmd UtilityOraclesqlplus64See theOracle Instant Clienthomepage for details. On Mac, followthese instructions. Thensudo ln -s /usr/local/bin/sqlplus /usr/local/bin/sqlplus64to make the binary accessible assqlplus64.SQLitesqlite3Available in standard distributions. Version >3.20.x required (not the case on Ubuntu 14.04).Big QuerybqSee theGoogle Cloud SDKpage for details.SnowflakesnowsqlSeeSnowSQL (CLI Client)DatabricksdbsqlcliIncluded when using package extradatabricksvia packagedatabricks-sql-cli. SeeDatabricks SQL CLIMake it so! Auto-migration of SQLAlchemy modelsAlembic has a featurethat can create a diff between the state of a database and the ORM models of an application. This feature is used inmara_db/auto_migrate.pyto automatically perform all necessary database transformations, without intermediate migration files:# define a model / tableclassMyTable(sqlalchemy.ext.declarative.declarative_base()):__tablename__='my_table'my_table_id=sqlalchemy.Column(sqlalchemy.Integer,primary_key=True)column_1=sqlalchemy.Column(sqlalchemy.TEXT,nullable=False,index=True)db=mara_db.dbs.SQLiteDB(file_name='/tmp/test.sqlite')# create database and tablemara_db.auto_migration.auto_migrate(engine=mara_db.auto_migration.engine(db),models=[MyTable])# -># Created database "sqlite:////tmp/test.sqlite"## CREATE TABLE my_table (#     my_table_id SERIAL NOT NULL,#     column_1 TEXT NOT NULL,#     PRIMARY KEY (my_table_id)# );## CREATE INDEX ix_my_table_column_1 ON my_table (column_1);When the model is changed later, thenauto_migratecreates a diff against the existing database and applies it:# remove index and add another columnclassMyTable(sqlalchemy.ext.declarative.declarative_base()):__tablename__='my_table'my_table_id=sqlalchemy.Column(sqlalchemy.Integer,primary_key=True)column_1=sqlalchemy.Column(sqlalchemy.TEXT,nullable=False)column_2=sqlalchemy.Column(sqlalchemy.Integer)auto_migrate(engine=engine(db),models=[MyTable])# -># ALTER TABLE my_table ADD COLUMN column_2 INTEGER;## DROP INDEX ix_my_table_text_column_1;Use with care! The are lot of changesthat alembic auto-generate can not detect. We recommend testing each aut-migration on a staging system first before deploying to production. Sometimes manual migration scripts will be necessary.Installationpipinstallmara-dborpipinstallgit+https://github.com/mara/mara-db.gitOptional: Installation of requirements for SQL ServerFor usage with SQL Server, the python module pyodbc and a odbc driver (e.g. Microsoft ODBC Driver 17 for SQL Server) is required which is not included in the general requirement.To see how to install pyodbc, take a look intothis install guide.
To see how to install ODBC 17, take a look intoInstalling the Microsoft ODBC Driver for SQL Server on Linux and macOS.On Linux, you most likely will have to deal with an SSL issue, seethis issue. A quick, dirty option in a test/development environment could be todisable the requirement for TLS 1.2.Optional: Installation of requirements for BigQueryFor usage with BigQuery, the officialbqandgcloudclients are required.
See theGoogle Cloud SDKpage for installation details.Enabling the BigQuery API and Service account JSON credentials are also required as listed
in the official documentationhere.One time authentication of the service-account used:gcloud auth activate-service-account --key-file='path-to/service-account.json'Optionally, for loading data from files into BigQuery, thegcloud_gcs_bucket_namecan be specified in the database initialization.
This will use the Google Cloud Storage bucket specified as cache for loading data and over-coming potential limitations.
For more seeloading-data.
By default, files will directly loaded locally as described inloading-local-data.A BigQuery context with a python cursor is also available on demand for easy access to BigQuery databases.
In order to use, install the official Google python client library:google-cloud-bigquery.LinksDocumentation:https://mara-db.readthedocs.io/Changes:https://mara-db.readthedocs.io/en/latest/changes.htmlPyPI Releases:https://pypi.org/project/mara-db/Source Code:https://github.com/mara/mara-dbIssue Tracker:https://github.com/mara/mara-db/issues