CTEs & Optimization
By: Antoine Lizée
Tags: Postgres, Data, Engineering, Sql, Postgresql

Photo by Susan Holt Simpson on Unsplash

This is a “technical” post that shares insights on a specific problem.

The content is specific to PostgreSQL, an open-source technology for relational databases that we use (and love) at Alan.

CTEs are NOT sub-queries

CTE stands for “Common Table Expression”. They are the SQL constructs that allow you to create an “intermediary table” that you can then query later:

CTEs are a key tool to write SQL code

You may love CTEs for their readability. Processing steps are exposed linearly, in order, instead of the encapsulated vision you get with sub-queries. Names are first, like defining variables in a normal programming language. At Alan, we surely do love them and use them everywhere!

Yet, improved readability comes with a few side effects. CTEs have been originally implemented with an optimization intent. They are a way to tell the query planner “Materialize this intermediate table, because I will use it several times in the future. Please don’t recompute it each time”. It’s indeed positive in a few rare cases, like below.

If getting the country is expensive, materializing the intermediate table is an upside

Potential Dire Consequences

The materialization of CTEs actually brings problems in most cases. Compared to using sub-queries, there are a few key downsides:

Indexes cannot be used efficiently after a CTEs, in JOINs or WHERE clauses for instance

All columns of a CTE are computed, even if they are not used in the rest of the query¹

Materialization itself can be a bit costly (usually insignificant though)

These are usually not a big deal, but you need to be aware of them if you are trying to make faster a query that has CTEs in it.

PostgreSQL 12: a mini revolution

Postgres is an amazing project that is not afraid to change something quite central in their syntax if that goes in the right direction. See below the 4th line of the release notes for Postgres 12:

…with more explanation a bit lower:

What does that mean?

By default CTEs now behave like sub-queries (“inlining”). As the result, they are mainly recognized as a semantic tool to organize your queries.

to organize your queries. They keep the advantages of a CTE and are automatically materialized if they are used several times. AKA: just chill out, Postgres is taking care of it.

If you want to force or prevent materialization, you can specify the keywords in the CTE syntax directly (see below for examples).

An application: foreign data wrappers (“fdw”)

Postgres proposes the powerful feature to directly query a remote database as a different schema. This is very handy, but can work poorly if misused.

First, we want to share a classic example where the use of a CTE can pose a problem with a remote server queried by a local server (pre-PG12) through a fdw. If you materialize some of the data first, you bring “back” in the local server this data. Then, any join or further processing with this data will need to happen in the local server, requiring to transfer huge payloads of data and preventing the use of indexes available in the remote server. In our experience, using a subquery instead of a CTE in those cases can lead to huge gains.

On PG<12, using a CTE might take 46min instead of 6 seconds with a subquery

Second, we want to share an example where CTEs can actually solve an issue with a fdw. If you have a query that uses a function from the local server in the WHERE clause, it will likely prevent JOINs to be done remotely. A typical example is the use of current_date for which the local server cannot let the remote server "decide" since the timestamps of the machines might be different. In such cases, the CTE can provide much needed help by forcing materialization and letting the joins happen in the remote server as a first step:

CTEs to the rescue: we split the computation steps and keep the joins remote

Conclusion

If you’re on Postgres < 12, you should still use CTEs, because readability gains outweigh the occasional optimization problems. Yet, you should know the limits of using CTEs when debugging long queries. Also, consider upgrading Postgres to a more recent version!

If you’re on Postgres >= 12, congrats: CTEs will not induce a performance hit anymore. You should still know when and how to use the keywords (NOT) MATERIALIZED, as they can help for optimization in some corner cases.

In general, you should rely on the documentation of Postgres. It can be very difficult to find the information you want (use google), but it’s always complete and clear. The answer to your problem will 99% be in the doc.

If you’ve read that far, you’re going deep in the technology you use to get the impact you want. We do too! If you want to join a team that uses technology to improve health, drop us a line at jobs@alan.eu