Jinja + SQL = ❤️. Macros for maintainable, testable data... | by Naim Kabir | Towards Data Science

SQL is an analyst’s bread and butter. It’s powerful, expressive, and flexible — but the more power a language gives you, the more ways you have of shooting yourself in the foot.
Better abstractions can help us with this. If we can abstract away bits of code that we use all the time, then we only have to write and check that code once. With the right abstractions, you can also start thinking about unit testing SQL, which isn’t done enough in professional analytics but should be standard practice.
Jinja macros are an especially good fit for abstracting over SQL — and will also set you up for success should you decide to use something like Fishtown Analytics’ Data Build Tool for your data organization — so we’re going to explore using Jinja specifically. But the lessons here should be generalizable beyond any one specific tool.
You can follow along with a few toy use-cases I’ve included in this Git repo. Code snippets in this article that correspond with a file in the repo will be headed with something like:
-- path: macros/volume_by_demographic.sql
Here’s a scenario: let’s say you have a hub-and-spoke model with a users table that links out to various other tables, like purchases, sessions, marketing_pushes, etc.
You may want to find out how volumes of purchases vary by demographic. That query could look like this:
-- Get purchase volume by demographicselect    users.age_group,    users.region,    count(*) as purchase_volumefrom purchases    join users on purchases.user_id=users.user_idgroup by 1,2
Pretty simple so far. But what if we wanted a count of user sessions cut by the same dimensions?
We would do:
-- Get session volume by demographicselect    users.age_group,    users.region,    count(*) as session_volumefrom sessions    join users on sessions.user_id=users.user_idgroup by 1,2
That’s a lot of duplicated code! Why write the same thing twice when we could instead have an abstraction that encodes basic assumptions about your data models?
Let’s write a macro. These macro snippets have macro logic in bold, where the variable names get replaced with a string value in the final “render” of a query when the macro is called. To understand the syntax better, check out the Jinja Template Designer docs.
-- path: macros/volume_by_demographic.sql{% macro volume_by_demographic(entity_table, count_col_name) %}select    users.age_group,    users.region,    count(*) as {{ count_col_name }}from {{ entity_table }}    join users on {{ entity_table }}.user_id=users.user_idgroup by 1,2{% endmacro %}
With this macro, those last two queries we made for purchase volumes and session volumes can be fully rendered from simple calls:
-- path: models/purchase_volumes.sql{% from 'volume_by_demographic.sql' import volume_by_demographic %}-- Get purchase volume by demographic{{ volume_by_demographic('purchases', 'purchase_volume') }}
Or:
-- path: models/session_volumes.sql{% from 'volume_by_demographic.sql' import volume_by_demographic %}-- Get session volume by demographic{{ volume_by_demographic('sessions', 'session_volume') }}
Try it out in the repo, and watch how values are swapped in to render final queries!
>> ./render.py models/session_volumes.sql
Now what if there’s a business-wide decision that the notion of ‘demographic’ shouldn’t just be age-group and region, because ‘region’ is too coarse to make any actionable decisions?
Well you can just go back into the macro code and change ‘region’ to ‘state’ or what-have-you, and leave the ultimate call-sites of the the macro unchanged. You only need to make the change in one place.
What you’re doing with abstractions is implementing an interface for business analytics, where you can swap out backend implementation and leave consumers of your analytics logic unaffected — leaving you with a SQL codebase that is easy to change with dynamic requirements.
Building interfaces is tricky, as we’ll see (we’ve already made a few mistakes, it turns out) — but starting somewhere is better than having no interfaces at all.
SQL allows you to nest logic (even recursively) in subqueries or common table expressions, letting you link one set of records to another to another ad infinitum.
Queries like this can get hairy, even with a perfect toy example like we’re exploring. For example, let’s say that we wanted to relate the volume of sessions to the volume of purchases for each of our demographic groups, to get a picture of how many purchases you can expect from each session.
In raw SQL, this might look like:
with purchase_volumes as (select    users.age_group,    users.state,    count(*) as purchase_volumefrom purchases    join users on purchases.user_id=users.user_idgroup by 1,2),session_volumes as (select    users.age_group,    users.state,    count(*) as session_volumefrom sessions    join users on sessions.user_id=users.user_idgroup by 1,2)-- Get purchase/session ratios for each age_group*state combinationselect     age_group,    state,    purchase_volume / session_volume as ratiofrom purchase_volumes     join session_volumes    on purchase_volumes.age_group=session_volumes.age_group    and purchase_volumes.state=session_volumes.state
That’s a lot of code, and each item we declare is a potential spot where an error can sneak in. It is provably true that complexity makes systems like a SQL-query based decision process more sensitive to small errors, and that bears itself out in the world of professional analytics often.
Abstractions collapse that surface area for bugs. The query above can be distilled down into:
-- path: models/purchase_sessions_ratio.sql-- Get purchase/session ratios for each age_group*state combinationselect     age_group,    state,    purchase_volume / session_volume as ratiofrom (        {{ volume_by_demographic('purchases', 'purchase_volume') }}     ) p       join (        {{ volume_by_demographic('sessions', 'session_volume') }}    ) s    on p.age_group=s.age_group    and p.state=s.state
Much less to think about!
Now we can focus on just the conversion ratio logic while trusting that the basic volume count logic is correct. Fewer moving pieces = fewer broken systems.
Try it in the repo:
>> ./render.py models/purchase_sessions_ratio.sql
Abstractions let you get even more flexible than what we’ve already built. What if instead of looking at the ratio of sessions to purchases, we actually want to look at the ratio of marketing pushes to sessions — and also the ratio of marketing pushes to purchases?
We could just macro-fy that last block of SQL we worked on so it takes different table references as parameters! We’ll have a numerator_vol_ref that specifies which table’s count should be the numerator of the output ratio, and a denominator_vol_ref that specifies the same for the denominator.
-- path: macros/volume_ratio.sql{% macro volume_ratio(    numerator_vol_ref,     denominator_vol_ref) %}select     age_group,    state,    N / D as {{ numerator_vol_ref }}_over_{{ denominator_vol_ref }}from (        {{ volume_by_demographic(numerator_vol_ref, 'N') }}     ) p       join (        {{ volume_by_demographic(denominator_vol_ref, 'D') }}    ) s    on p.age_group=s.age_group    and p.state=s.state{% endmacro %}
Now if we wanted to compute purchase-to-session ratios again, we could ask:
-- path: models/purchase_sessions_ratio_2.sql{%- from 'volume_ratio.sql' import volume_ratio -%}-- Get ratio of purchases to sessions for a demographic group{{ volume_ratio('purchases', 'sessions')}}
But what if we wanted to look at a whole bunch of steps in a funnel, and that list of steps is dynamic?
Enter: Jinja control flows. We can do loops and if-conditionals in our SQL templates, which is a superpower.
Let’s make a macro that references that last block of logic, but let’s take a denominator_vol_ref and a whole array of numerator_vol_refs as parameters.
-- macros/volume_ratios.sql{% macro volume_ratios(numerator_vol_refs, denominator_vol_ref) %}select    age_group,    state,    {%- for ref in numerator_vol_refs -%}    {{ ref }}_over_{{ denominator_vol_ref }}    {%- if not loop.last -%},{%- endif -%}    {% endfor %}from    ({{ volume_ratio(numerator_vol_refs.0, denominator_vol_ref) }})    as base    {% for ref in numerator_vol_refs %}    {% if not loop.first %}    join    ({{ volume_ratio(ref, denominator_vol_ref) }}) as {{ref}}_r            on {{ref}}_r.age_group=base.age_group            and {{ref}}_r.state=base.state    {%- endif -%}    {%- endfor -%}{% endmacro %}
There’s a lot going on there. Let’s break down some of what we’re doing.
In this block:
select    age_group,    state,    {%- for ref in numerator_vol_refs -%}    {{ ref }}_over_{{ denominator_vol_ref }}    {%- if not loop.last -%},{%- endif -%}    {% endfor %}
We’re looping through the list of table references whose demographic-wise volumes we’ll use as ratio numerators. The if not loop.last conditional statement makes sure we stick a comma after each column name except the last one, to get well-formatted SQL.
This gives us a set of references to column names that are exposed in subqueries.
In this block:
from    ({{ volume_ratio(numerator_vol_refs.0, denominator_vol_ref) }})    as base
We create a volume ratio subquery from the first numerator table reference using a .0 index into the array, and alias that subquery as base.
In the last block:
{% for ref in numerator_vol_refs %}    {% if not loop.first %}    join    ({{ volume_ratio(ref, denominator_vol_ref) }}) as {{ref}}_r            on {{ref}}_r.age_group=base.age_group            and {{ref}}_r.state=base.state    {%- endif -%}{%- endfor -%}
We iterate again through the list of numerator table references. Since we already used the first table reference to create the base subquery we ignore the first one with an if not loop.first conditional.
Then we join base against new volume_ratio subqueries programmatically for each numerator table reference, eventually exposing all the columns we want.
We can call the macro in a file:
-- path: models/example_funnel.sql{%- from 'volume_ratios.sql' import volume_ratios -%}{{ volume_ratios([        'purchases',         'sessions',         'supportTickets'        ],    'marketing_pushes') }}
Try it in the repo:
>> ./render models/example_funnel.sql
That thing will render a gigantic beast of a query that is hard to understand and would have been a pain to put together by hand. But with macros? Not terribly terrible.
Here’s the real thesis of this whole piece.
We’ve seen how abstractions let us compose long complicated queries by composing little Lego pieces. If you use them in your workflow you can build something that is ultimately complicated from relatively simple parts.
We’ve also seen that these pieces are parametrizable, and that a key parameter we can feed macros is a reference to a table! This reference can be to a “real” materialized database table, a common table expression, or a view. What’s important is we can use a parametrized query on both production data or a synthetic set of data we’ve intentionally devised and stored somewhere.
This gives us the power to write a bunch of rows that cover a whole gamut of test cases we want our queries to deal with, and check that the results are as expected. Then, we can use that very same query logic on actual business-relevant quantities once we’re certain it works.
If we engaged in some unit testing with the code we’ve already explored, we might have discovered a few bugs already. For example, in volume_ratio.sql, we’re doing integer divisions and won’t be getting real numbers in our query results. We also don’t check for 0-valued denominators, which would be bad news in the worst case.
There are also several cases where we make implicit assumptions about how exposed columns will be named — and any slight change to the naming scheme will break downstream macros without callers knowing what happened.
Doing unit-tests stresses the whole analytics house of cards your organization is building and allows you to iterate until you get a “Data API” that is both correct and easy to change.
It also allows you to confidently make changes and simply check for regressions afterwards, instead of spending your time waist-deep in production records trying to do sanity checks.
The lessons from this article are only going to be useful if you mold your analytics team a particular way. Some implicit assumptions that must be true before you can even start using Jinja2 and the abstraction magic of macros:
You need to encode analytics work in a unified codebase, not in a smattering of Business Intelligence tool pages across a handful of teams.
There should be a continuous “deploy” pipeline that takes the latest SQL from the codebase and actually shunts it into dashboards/reports/etc. This pipeline can handle Jinja rendering, query export, etc.
There must be a framework to actuate tests and stop the line if there’s an assumption or logic failure that could produce incorrect query results.
A great way to get started with all of this is to go through documentation for Fishtown Analytics’ Data Build Tool, which was designed to handle a lot of the nuts and bolts for an organization like this.
Going down this path will elevate your analytics organization beyond a crew of detectives, ad-hoc query handlers, and storehouses of tribal institutional knowledge.
It will turn them into a team of analytics engineers, whose work is repeatable and reliable.
Jinja2 Template docs
Data Build Tool tutorial