Climbing B-tree Indexes in Postgres | by GreekDataGuy | Towards Data Science

Your query was slow so you decided to speed it up by adding an index. What type of index was it? Probably a B-tree.
Postgres has a few index types but B-trees are by far the most common. They’re good for sorting and matching; and once you understand what they look like under the hood, it’s immediately apparent why.
We’re going to take a dive into the internals of B-tree’s implementation in Postgres, then do some SQL examples to show their effect. I’ve provided queries so you can run the same experiments yourself.
This post assumes you already have a general idea what an index is and does. If not, the often provided abstraction is the glossary in a textbook. Rather than reading every line in the book to find a word/topic, you can look it up alphabetically in the glossary, then jump to the page it’s on. We’ll see shortly the reality is slightly more complicated.
B-tree stands for Balanced Tree.
It’s NOT a binary tree, which allows a max of 2 child nodes per parent and was designed for in-memory search.
From Wikipedia,
In computer science, a B-tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time. The B-tree generalizes the binary search tree, allowing for nodes with more than two children.[2] Unlike other self-balancing binary search trees, the B-tree is well suited for storage systems that read and write relatively large blocks of data, such as discs. It is commonly used in databases and file systems.
That’s a pretty good non-technical introduction. For a really technical explanation, see this paper by Lehman and Yao, on which the Postgres implementation is based.
B-trees are “balanced” because the distance from the root to any leaf node is the same. A leaf node is a node without children. The root node is the node at the top.
A node has keys. In our root node below, [10,15,20] are keys. Keys map to values in the database but also bound keys in child nodes.
The first child node, [2, 9], has values less than 10, hence the pointer on the left side of 10.
The 2nd child node, [12], has a value between 10 and 15, hence the pointer from there.
The 3rd child node, [22] is greater than 20, hence the pointer on the right side of 20.
Now if we wanted to look up the key, 12, we’d compare 12 to values in the root node and see that it’s between 10 and 15. So we’d use the pointer between 10 and 15 to find the node containing 12.
This is an abstraction over Postgres’ implementation but you can already imagine why this is faster than iterating over every number in a table and checking if it equals 12.
This is why B-trees can search, insert and delete in O(logN) time.
B-trees also have a minimum and maximum number of keys per node. Nodes are joined and split on inserts and deletes to keep nodes in range.
We’re going to work through 3 examples where B-trees can have a huge impact: numbers, text and dates.
Create a table for each datatype. Each table has 2 columns but we’ll only index one of them.
create table numbers(  val integer,  val_indexed integer);CREATE INDEX numbers_indexed ON numbers using btree (val_indexed);create table strings(  val varchar,  val_indexed varchar);CREATE INDEX strings_indexed ON strings using btree (val_indexed);create table dates(  val varchar,  val_indexed varchar);CREATE INDEX dates_indexed ON dates using btree (val_indexed);
Let’s generate 1M random integers between 1 and ~1M.
insert into numbersselect  (random()*1000000 +1)::int,  (random()*1000000 +1)::intfrom (SELECT * FROM generate_series(1,1000000)) ser;
We’ll repeatedly run the below queries, record query time, then add an additional 1M additional records.
select * from numbers order by val desc;select * from numbers order by val_indexed desc;
What we see is a massive difference in query time in ordering the indexed vs non-indexed column.
5 million records is small potatoes for Postgres but we can already see a huge difference in efficiency.
Why the difference? Let’s use the explain function in Postgres to dig into it.
explain select * from numbers order by val desc;explain select * from numbers order by val_indexed desc;
We see the non-indexed search using a sequential scan, while the indexed search using an index scan. Given the above discussed structure of a B-tree, it’s easy to see why searching the index would be much faster than searching the table.
Let’s do the same comparison but with strings. Though this time we’ll incrementally add 100k rows at a time.
insert into stringsSELECT  md5(random()::text),  md5(random()::text)from (  SELECT * FROM generate_series(1,100000) AS id) AS ser;
Now order those rows.
select * from strings order by val desc;select * from strings order by val_indexed desc;
What do we see? The exact same pattern where the indexed search blows non-indexed search out of the water.
Peeking at explain , we see the same cause.
explain select * from strings order by val desc;explain select * from strings order by val_indexed desc;
Let’s do this one more time with dates. Again, we’ll only generate 100k rows at a time.
with cte as (select   timestamp '1900-01-10 20:00:00' +  random() * (    timestamp '2000-01-20 20:00:00'-timestamp '1900-01-10 10:00:00'  ) rdate  from (SELECT * FROM generate_series(1,100000) AS id) ser)insert into datesselect   rdate,  rdatefrom cte;
Query the dates and record query times.
select * from dates order by val desc;select * from dates order by val_indexed desc;
Compare the difference.
We can already guess what explain is going to show but let’s take a look anyway.
explain select * from dates order by val desc;explain select * from dates order by val_indexed desc;
Once again, ordering rows via index scans is orders of magnitude faster than sequence scans.
My aim was to give you a general sense of how indexes are structured so you can intuitively understand why they improve query times.
Then we covered some examples where adding an index made a night-and-day difference.
We only covered ordering rows, but it would be interesting to investigate query times and look at explain for other types of queries like where or exists.