dbt at GitLab
How we use one of the most popular Data
tool
Radovan Bacovic
Staff Data Engineer @ GitLabWe are using our product which we areusing to build our product to build ourdata product to analyze the data fromour product.

Confusing?

????????

This looks like recursion

Stco illnfusing

We are using our product which we areusing to build our product to build ourdata product to analyze the data fromour product.

We are using our product which weare using to build our product tobuild our data product to analyze
the data from our product.

We are using product GitLab tobuild data product (EnterpriseData Platform)
to analyze the data from GitLabAbout GitLab (company)

What makes GitLab unique?

What makes GitLab unique?
(Probably) the most transparent company in the worldWhat makes GitLab unique?
Total transparent operational model - so transparent that
it is intimidating

What makes GitLab unique?
All remote from the beginning

What makes GitLab unique?
The biggest all-remote company in the worldWhat makes GitLab unique?
All-remote and async work advocate

What makes GitLab unique?
A global leader in distributed work

What GitLab is doing?

What GitLab is doing?
The DevSecOps Platform delivered as a single
application to help you iterate faster and innovate
togetherGitLab by num83r5

0We have 0 offices1600
1600+ team members in more than 65 countries andregions30+ millions
30+ million registered users

22Releases a new version of the product on the 22nd of
every month.

2000
GitLab handbook has over 2000 web pages of text.

Let’s jump in the dbt machinery

What is dbt?

GitLab dbt use case(mainly) open source tech stack(mainly) open source tech stackWhy dbt?

1. Open source

Why dbt?

1. Open source
2. Easy to learn

Why dbt?

1. Open source
2. Easy to learn
3. Strong community

Why dbt?

1.
2.
3.
4.

Why dbt?

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation

1.
2.
3.
4.

Why dbt?

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation
5. Portable

1.
2.
3.
4.

Why dbt?

Open source
Easy to learn
Strong communitySingle Source of truth foryour transformation
5. Portable
6. Build in Python + Jinja +SQL

1.
2.
3.
4.

Why dbt?

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation
5. Portable
6. Build in Python + Jinja +SQL
7. Usage: Jinja + SQL

1.
2.
3.
4.

Why dbt?

5.
6.
7.
8.

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation
Portable
Build in Python + Jinja +SQL
Usage: Jinja + SQL
A plenty of externallibraries

1.
2.
3.
4.

Why dbt?

5.
6.
7.
8.
9.

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation
Portable
Build in Python + Jinja +SQL
Usage: Jinja + SQL
A plenty of external
libraries
Support for Python models
and ML1.
2.
3.
4.

Why dbt?

5.
6.
7.
8.
9.
10.

Open source
Easy to learn
Strong community
Single Source of truth foryour transformation
Portable
Build in Python + Jinja +SQL
Usage: Jinja + SQL
A plenty of external
libraries
Support for Python models
and ML
Extensive

GitLab architecture for dbt

GitLab architecture for dbt

PREPRAW

1.
2.
3.

4.

Unﬁltered data
Raw format
Various formats(structured,semi-structured data)
Result of ingestion byvarious tools - depends
on the use case

1.
2.
3.
4.

Clean data
Formatted data
Validated data
Result oftransformation by dbt

PROD

1.
2.
3.
4.

Sensitive data
Hashed data
Separated per function
Result oftransformation by dbt

GitLab architecture for dbt

RAW

Used as a source for thetransformation

1.
2.
3.
4.
5.

PREP

PROD

Cleaned up data
Hashed data
Filtered
Aggregated data
Legacy data

Data ready for:1.
Visualisation
2.
Data Science
3.
Further use
4.
Modeled data using
dimensional modeling

GitLab architecture for dbt

Best practices
Development setup

Best practices
MacrosBest practices
Tags

Best practices
Selectors

Best practices
SourcesBest practices
Sensitive data

Best practices
Sensitive data- hashing

Best practices
Sensitive data
- Dynamic masking

Best practices
Warehouse size

Best practices
Testing

Best practices
Testing

Best practices
Testing

Automation

Automation
Pipelines - testingAutomation
Pipelines - testing● Clone environments

Automation
Pipelines - testing● Clone environments● Run models

AutomationPipelines - testing● Clone environments● Run models● SAFEStay on the bright side - SAFE framework●●●●Sensitive
Accurate
FinanceEffectAutomation
Pipelines - testing● Clone environments● Run models● SAFE● Linters

CloneRun

Misc / testing

Documentation

Automation
Pipelines - production

Automation
Documentation
https://dbt.gitlabdata.com/

Let's wrap up
andgather takeaways

Speaking about transparency through ourcultural lens, we are:“Short Term Critical And Long Term
Optimistic”

dbt is the proper fit for ourtransformation use caseSSOT (Single source of truth) for ourneeds regarding the data project

Stick with GitLab philosophy about
DevOps and Open Source

Allows us better and integrative
collaboration among teams

About me
Find me, ping me, ask meThank you!