Here Is How You Can Apply Software Development Best Practices to Analytics Pipelines
By: Tanmay Deshpande
Tags: Tech, Software Development, Programming, Technology, Data

Configuring the Big Query Credentials

You can configure the Big Query credentials using the methods described on the dbt site. Here I am using the service account JSON key method. You need to make sure the service account has following permission —

BigQuery Data Editor

BigQuery Job User

BigQuery User

Then you need to make sure dbt profile is setup $HOME/.dbt/profile.yml and details for the service account are updated as shown below —

my-bigquery-db: target: dev outputs: dev: type: bigquery method: service-account project: my-project dataset: olympics threads: 1 keyfile: /path/to/service-account.json timeout_seconds: 300 priority: interactive retries: 1

You can set up profiles for dev and prod separately without affecting any code change. This one of the features of dbt which allows us to do the sandboxing and environment management easily.

Creating a Project

You can create a dbt project using a simple CLI command

$ dbt init [project-name]

This command creates a sample dbt project with all the required files and folders. The typical structure looks like as shown below -

Looking at the folder structure, you can see the sense of software development principles where you see folders for tests, logs, modules, data separately. This allows the overall manageability and readability of the code structure. This is very much version control friendly and easy to follow.

You can read about the details of the folder structure and their meaning on the dbt site.

In dbt_project.yml , you need to set up the profile you configured in the earlier section.

Creating Models

I am creating two simple models to manipulate the data and get some insights.

The models/example/athletes.sql looks like this —

{{ config(materialized='table') }} with athletes as ( select * from olympics.atheletesnew )

select * from athletes

and model/example/players_by_country.sql looks like this —

{{ config(materialized='table') }} with athletes as ( select * from {{ ref('athletes') }} ), players_by_country as ( select country, count(*) as totalcount from athletes group by country ) select * from players_by_country

Here you can see the dbt feature where you can define a model in one file and refer it in other models. This helps in re-usability. Also if you need to change anything in base model, you can do it in one place. Just like in any typical software engineering language.

Models have other features like —

Enabling / Disabling Model with flag

Using aliases

Using variables

Using tags

etc.

You can create models inside the specific folders. This allows you to achieve modularity. This is similar to creating packages/namespaces in certain programming languages.

Documention

dbt allows you to auto-generate documentation for the models. Making it easier for the team to understand and collaborate better

You can generate docs by running a command

$ dbt docs generate

You will be able to look at the documentation by running a command —

$dbt docs serve

You can even look at the lineage information in the documentation website as shown below —

Testing

dbt allows you to do some testing of the models generated. By default, you can configure tests like a unique check, null check, referential integrity etc. for a model. In this example, I am running sample schema tests to validate the results of the model for null checks. I can configure these tests in schema.yml as shown below —

You can run tests for this project by running a simple command like

$dbt test

and see output like this —

You can even run some complex tests as described on the dbt site.

Logs

Every time you compile and run a dbt project, it will generate the detailed logs for you. In case, you face any issues, these logs are quite useful to trace back the errors.

Integrating with CI/CD Pipelines

All dbt commands give proper exit codes like 0, 1, 2. You can use this feature in a CI/CD pipelines so that you know if a particular step was successful or failed.