Tuning your PostgreSQL for High Performance
By: Luis Sena
Tags: Database, Postgresql, Sql, Development, Postgres

Table of Contents

Common Issues

General principles

Indexes

Index Types

Improving queries

Locks

Rules of thumb

PG Config

BULK Updates/Inserts

Assumptions

You know basic SQL

You’ve already used PostgreSQL in the past

Basic knowledge of what an index and constraints are

Common issues you might have faced in the past

A query is slow periodically

A query is slow at times, but only affects certain users

High memory usage

High query latency, even for simple queries

The database is not responding

My server code is not able to connect but I’m able to connect with my root account

General principles

Why use a subdomain instead of IP/Hostname of your database

This will allow you to hot-swap servers with less chance of downtime, de-commission nodes, simple round-robin, and geo-distribution to name a few examples.

When your database is smaller than 10 GB

if your database has less than 10 GB. It will be fast with the default settings unless you really start to go crazy with your queries and that’s usually a good sign that your table layout needs to be improved

Read Replicas. Why and How

Postgres is an amazing database, but it has one “problem”. The Vanilla version can only have one write node.

This means there is only so high you can vertically scale your machine.

A simple way to overcome this is by adding read replicas and only use your write instance for writes and high latency sensible queries.

This will also help you with lock/deadlock issues since you will be splitting your reads and writes (caution when you expect those locks to save you from inconsistent behavior on reads)

Use Read Replicas to increase your throughput and resilience

PGBouncer + Connection Pooling

PostgreSQL was built to handle a small number of long-lasting connections.

That’s why it has a relatively small default limit of connections and it’s not uncommon for people to get the error “connection limit exceeded for non-superusers”.

One might be tempted to just increase that limit, but there are better ways.

Postgres will usually complete the same 10,000 transactions faster by doing them 5, 10, or 20 at a time than by doing them 500 at a time. Determining exactly how many should be done at once varies by workload and requires tuning.

https://wiki.postgresql.org/wiki/Number_Of_Database_Connections

https://www.cybertec-postgresql.com/en/tuning-max_connections-in-postgresql/

Another thing to take into consideration with connections is that creating and destroying them sometimes can be responsible for as much as half of that operation time and resource usage.

Because of that, it’s always recommended to have persistent connections.

This, in turn, will just exacerbate the previous limit issue.

So how can we solve this?

The answer is using a connection pool. Sometimes the framework you’re using offers that (like SQLAlchemy), other times you might need a specialized tool.

But if you’re using multiple machines to serve your API (as you probably are), you will run into the previous problem again since your framework pooler will only be responsible for managing the instance it’s running on.

When you have multiple servers, a good solution is using PGBouncer.

This amazing tool will allow you to connect multiple servers to it instead of connecting directly to Postgres.

It will queue incoming connections when the concurrent amount is greater than the one specified by you.

I’ll leave PGBouncer for another post.

PosgreSQL performance will increase dramatically if you have a small number of long-lasting connections instead of constantly creating new ones.

pgBadger

pgBadger is a PostgreSQL log analyzer built for speed that is able to generate full reports from your PostgreSQL log file and show them on a web interface. It’s a single and small Perl script that outperforms any other PostgreSQL log analyzer.

I always recommend using it if your Postgres is critical to your systems.

Keep accurate logs and metrics of your servers so you can make informed decisions

Your transactional DB is not for logging

Logging using your database can lead to massive bloat and it will create a bottleneck on various fronts.

Use s3, cloudwatch, papertrail, elk, logz.io, etc.

Not only will it be faster, cheaper, and allow better visibility, it will prevent your logs from affecting your transactional queries.

OLTP vs OLAP

Pg is usually used in OLTP scenarios with a bit of OLAP.

That is fine as long as you understand the difference between them.

For example, if a public-facing endpoint contains a very complex query that loads a lot of data and does complex operations, it’s probably a good indicator that it should be treated as OLAP and that endpoint refactored to adopt a different strategy.