Package: dbd
dbd: database prototyping tooldbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.dbd helps you with following tasks:Loading CSV, JSON, Excel, and Parquet data to database. It supports both local and online files (HTTP URLs). Data can be loaded incrementally or in full.Transforming data in existing database tables using insert-from-sql statements.Executing DDL (Data Definition Language) SQL scripts (statements likeCREATE SCHEMA, etc.).How dbd worksdbd processes a model directory that contains directories and files:Directoriescreate new database schemas.Filescreate new database table or view. The new table's or view's name is the same as the data file name..csv,.json,.xlsx, and.parquetdata files are introspected and loaded to database as tables..sqlfiles that contain SQL SELECT statements are executed and the result is loaded to database as table or view..reffiles contain one or more local paths or URLs pointing to supported data files. The referenced files are loaded to database as tables..yamlfiles contain metadata for the files above. The.yamlfile has the same name as a data,.sql, or.reffile and specifies details of target table's columns (data types, constraints, indexes, etc.)..yamlfiles are optional. If not specified, dbd uses defaults (e.g.TEXTdata types for CSV columns).ddlfiles contain multiple SQL statements separated by semicolon that are executed against the database.dbd knows the correct order in which to process files in the model directory to respect mutual dependencies between
created objects.dbd currently supports Postgres, MySQL/MariaDB, SQLite, Snowflake, BigQuery, and Redshift databases.Getting started and ExamplesA short 5-minute getting started tutorial is availablehere.You can also check out dbd'sexamples here.
The easiest way how to execute them is to either clone or download dbd's github repository and start with theSQLite examples.python3-mvenvdbd-envsourcedbd-env/bin/activate
pip3installdbd
gitclonehttps://github.com/zsvoboda/dbd.gitcddbd/examples/sqlite/basic
dbdrun.These commands should create a newbasic.dbSQLite database witharea,population, andstatetables that are created and loaded from the corresponding files in themodeldirectory.Installing dbddbd requires Python 3.8 or higher.PrerequisitesCheck that you have a recent version of Python 3.8 or higher.python3-Vif not use a package manager to install the latest python:On Fedora run:sudoyuminstallpython3On Ubuntu run:sudoaptinstallpython3Install Python virtual environment:On Fedora run:sudoyuminstallpython3-virtualenvOn Ubuntu run:sudoaptinstallpython3-venvOn Windows just install Python 3.8 or higher from the Store.Then activate the virtual environment:On Linux run:python3-mvenvdbd-envsourcedbd-env/bin/activateOn Windows run:python3-mvenvdbd-env
calldbd-env\Scripts\activate.batPyPIpip3installdbdORgitclonehttps://github.com/zsvoboda/dbd.gitcddbd
pip3install.Running dbddbdinstalls a command line executable that must reside on your path. Sometimes Python places the executable
(calleddbd) outside of yourPATH. Try to executedbdafter the installation. If the command cannot be found,
try to executeexportPATH=~/.local/bin:$PATHand rundbdagain.pip3usually complains about the fact that the directory where it is placing the executable is
not inPATH. You need to take the scripts directory that it suggests and put it on yourPATH.Once you can execute thedbdcommand, clone the dbd repository and start with the SQLite examples:gitclonehttps://github.com/zsvoboda/dbd.gitcddbd/examples/sqlite/basic
dbdrun.You can also start withthis tutorial.Starting a new dbd projectYou can generate dbd project initial layout by executinginitcommand:dbdinit<new-project-name>Theinitcommand generates a new dbd project directory with the following content:modeldirectory that contains the content files.dbd.profileconfiguration file that defines database connections. The profile file is usually shared by more dbd projects.dbd.projectproject configuration file references one of the connections from the profile file and define themodeldirectory location.dbd profile configuration filedbd stores database connections in thedbd.profileconfiguration file. dbd searches for it in the current directory or in your home directory. You can use--profileoption to point it to a profile file in different location.The profile file is YAML file with the following structure:databases:db1:db.url:<sql-alchemy-database-url>db2:db.url:<sql-alchemy-database-url>db3:db.url:<sql-alchemy-database-url>Readthis documentfor more details about  specific SQLAlchemy database URL formats.dbd project configuration filedbd stores project configuration in project configuration file that is usually stored in your dbd project directory. dbd searches fordbd.projectfile in your project's directory root. You can also use the--projectoption of thedbdcommand to specify a custom project configuration file.The project configuration file also uses YAML format and references dbd model directory and databse connection from a profile config file. All paths in project file are either absolute or relative to the directory where the profile file is located.For example:model:./modeldatabase:db2Model directoryModel directory contains directories and files. Directories represent database schemas. Files, in  most cases, represent database tables.For example, thismodeldirectory layoutdbd-model-directory
+- schema1
 +-- us_states.csv
+- schema2
 +-- us_counties.csvcreates two database schemas:schema1andschema2and two database tables:us_statesinschema1andus_countiesinschema2. Both tables are populated with the data from the CSV files.dbd supports following files located in themodeldirectory:DATA files:.csv,.json,.xls,.xlsx,.parquetfiles are loaded to database as tablesREF files:.reffiles contain one or more absolute or relative paths to local files or URLs of online data files that are loaded to database as tables. All referenced files must have the same structure as they are loaded to the same table.SQL files:.sqlwith SQL SELECT statements are executed using insert-from-select SQL construct. The INSERT command is generated (the SQL file only contains a SQL SELECT statement)DDL files:contain a sequence of SQL statements separated by semicolon. The DDL files can be namedprolog.ddlandepilog.ddl. Theprolog.ddlis executed before all other files in a specific schema. Theepilog.ddlis executed last. Theprolog.ddlandepilog.ddlin the top-level model directory are executed as the very first or the very last files in the model.YAML files:specify additional configuration for theDATA,SQL, andREFfiles.REF files.reffile contains one or more references to files that dbd loads to the database as tables. The references can be URLs, absolute file paths or paths relative to the.reffile. All referenced data files must have the same structure as they are loaded to the same database table.Here is an example of a.reffile:https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/01-03-2022.csv
https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/01-04-2022.csv
../data/01-05-2022.csv
../data/01-06-2022.csvThe paths and URLs can point to data files with different formats (e.g. CSV or JSON) as long as the files have the
same structure (number of columns and column types).Referencing files inside ZIP archivesREF files support paths that reference files inside ZIP archives using the>path separator. For example:../data/archive.zip>covid-variants.csvORhttps://raw.githubusercontent.com/zsvoboda/dbd/master/tests/fixtures/capabilities/zip_local/data/archive.zip>covid-variants.csvKaggle datasetsYou can reference a kaggle dataset using thekaggle://kaggle-dataset-name>dataset-fileurl.For example:kaggle://kalilurrahman/new-york-times-covid19-dataset>us.csvKaggle authenticationTo use the Kaggle API, sign up for a Kaggle account athttps://www.kaggle.com. Then go to the 'Account' tab
of your user profile (https://www.kaggle.com/<username>/account) and select 'Create API Token'.
This will trigger the download of kaggle.json, a file containing your API credentials.
Place this file in the location~/.kaggle/kaggle.json(on Windows in the locationC:\Users\<Windows-username>\.kaggle\kaggle.json- you can check the exact location,
sans drive, withecho %HOMEPATH%).
You can define a shell environment variableKAGGLE_CONFIG_DIRto change this location to$KAGGLE_CONFIG_DIR/kaggle.json(on Windows it will be%KAGGLE_CONFIG_DIR%\kaggle.json).For your security, ensure that other users of your computer do not have read access to your credentials.
On Unix-based systems you can do this with the following command:chmod 600 ~/.kaggle/kaggle.jsonYou can also choose to export your Kaggle username and token to the environment:exportKAGGLE_USERNAME=datadinosaurexportKAGGLE_KEY=xxxxxxxxxxxxxxSQL files.sqlfile performs SQL data transformation in the target database. It contains a SQL SELECT statement that
dbd wraps in insert-from-select statement, executes it, and stores the result into a table or view that inherits its name from the SQL file name.Here is an example ofus_states.sqlfile that creates a newus_statesdatabase table:SELECT
        state.abbrev AS state_code,
        state.state AS state_name,
        population.population AS state_population,
        area.area_sq_mi  AS state_area_sq_mi
    FROM state
        JOIN population ON population.state = state.abbrev
        JOIN area on area.state_name = state.stateYAML files.yamlfile specifies additional configuration for a correspondingDATA,REForSQLfile with the same base file name. Here is a YAML configuration example for theus_states.sqlfile above:table:columns:state_code:nullable:falseprimary_key:truetype:CHAR(2)state_name:nullable:falseindex:truetype:VARCHAR(50)state_population:nullable:falsetype:INTEGERstate_area_sq_mi:nullable:falsetype:INTEGERprocess:materialization:tablemode:dropThis.yamlfile re-types thestate_populationand thestate_area_sq_micolumns to INTEGER, disallows NULL values in all columns, and makes thestate_codecolumn table's primary key.You don't have to describe all table's columns. The columns that you leave out will have their types
set to the default TEXT datatype in case of DATA files and is defined by the insert-from-select in case of SQL files.Theus_states.sqltable is dropped and data are re-loaded in full everytime the dbd executes this model.Table section.yamlfile's columns are mapped to a columns of the table that dbd creates from a correspondingDATA,REForSQLfile. For example, a CSV header columns or SQL SELECT columnAScolumn clauses.dbd supports following column's parameters:type:column's SQL type.primary_key:is the column part of table's primary key (true|false)?foreign_keys:all other database table columns that are referenced from a column in table (in formatforeign-table.referenced-column).nullable:does column allow null values (true|false)?index:is column indexed (true|false)?unique:does column store unique values (true|false)?Process sectionTheprocesssection defines following processing options:materialization:specifies whether dbd creates a physicaltableor aviewwhen processing  SQL file. TheREFandDATAfiles always yield physical table.mode:specifies what dbd does with table's data. You can specify valuesdrop,truncate, orkeep. Themodeoption is ignored for views.Iterative developmentThedbdtool's parameter--onlyhelps with iterative development process by allowing you to specify a subset of the
tables to process. The--onlyparameter accepts a comma-separated list of fully qualified table names (schema.table-name).
For example:dbd run --only stage.ext_countryonly processes theext_countrytable in thestageschema.The--onlyparameter also processes all passed table's dependencies. You can skip the dependencies with--no-depsoption.Jinja templatesMost of model files supportJinja2 templates. For example, thisREFfile loads 6 CSV files to database (4 online files from a URL and 2 from a local filesystem):{%forninrange(4)%}https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/01-0{{n+1}}-2022.csv{%endfor%}../data/01-05-2022.csv../data/01-06-2022.csvProfile an project configuration files also us Jinja2 templates. You can expend any environment variable with the{{ environment-variable-name }}syntax.
For example, you can define your database connection parameters like username or password in environment variables and use this profile configuration file:databases:states_snowflake:db.url:"snowflake://{{SNOWFLAKE_USER}}:{{SNOWFLAKE_PASSWORD}}@{{SNOWFLAKE_ACCOUNT_IDENTIFIER}}/{{SNOWFLAKE_DB}}/{{SNOWFLAKE_SCHEMA}}?warehouse={{SNOWFLAKE_WAREHOUSE}}"covid:db.url:"snowflake://{{SNOWFLAKE_USER}}:{{SNOWFLAKE_PASSWORD}}@{{SNOWFLAKE_ACCOUNT_IDENTIFIER}}/{{SNOWFLAKE_DB}}/{{SNOWFLAKE_SCHEMA}}?warehouse={{SNOWFLAKE_WAREHOUSE}}"Fast data loading modeAll supported database engines except SQLite support fast data loading mode. In this mode, data are loaded to a
database table using bulk load (SQL COPY) command instead of individual INSERT statements.MySQL and Redshift require additional configuration to enable fast data loading mode.
Without this extra configuration dbd reverts to slow inserting mode via INSERT statements.MySQLTo enable fast loading mode, you need specifylocal_infile=1query parameter in the MySQL connection url.
You also must enable the LOCAL INFILE mode on your MySQL server. You can for example do this by executing this
SQL statement:SETGLOBALlocal_infile=trueRedshiftTo enable fast loading mode, you need specifycopy_stageparameter in thedbd.projectconfiguration file.
Thecopy_stageparameter must reference a storage definition in yourdbd.profileconfiguration file.
Check the example configuration files in theexamples/redshift/covid_czdirectory. Here are the example definitions of the
environment variables that these configuration files use:exportAWS_COVID_STAGE_S3_URL="s3://covid/stage"exportAWS_COVID_STAGE_S3_ACCESS_KEY="AKIA43SWERQGXMUYFIGMA"exportAWS_COVID_STAGE_S3_S3_SECRET_KEY="iujI78eDuFFGJF6PSjY/4CIhEJdMNkuS3g4t0BRwX"Licensedbd code is open-sourced underBSD 3-clause license.Resources and Referencesdbd getting starteddbd github repodbd PyPiSubmit issue