Elixir, Ecto, Postgres: A Saga about Database Performance
By: Eric Scouten
Tags: Postgres, Elixir, Ecto

Initial Problem Statement

An application my team is building was missing our internal targets for write performance by a fairly wide margin. No matter what we tried, we kept bouncing off of levels between 500 and 1000 write requests per second in our deployed environment. We wanted to support substantially more than that. It turned out that this was a bit of a saga.

We’re sharing this here in the hopes that it saves somebody down the road from experiencing the same pain points. (Maybe you will have different ones, and if you’re able, maybe you will be kind enough to share your findings.) I’ll call out some of the data points that we observed and some of the theories that we had along the way, including those that turned out to be false alarms. I’m eliding some details in the interest of brevity. If you want the full play-by-play, I recommend reading the Elixir Forum thread.

This is a fairly typical Elixir database application. We’re fairly conservative about applying version updates, but not too far behind:

Elixir 1.8.1

Ecto 3.0

Postgrex 0.14

Postgres 10.9

Local dev environments: Various MacBook Pro laptops

Deployed environments: Amazon EC2 and RDS

Round 1: Attempt to Scale

We had started with fairly modest Postgres configuration ( db.t2.medium as I recall) and so our first thought was, “Can we throw more resources at it and make it better?” So we scaled up the number of server host instances from 2 to 4 and scaled up the RDS instance to a beefier size, eventually settling on a db.t3.2xlarge .

Increasing the server cluster size yielded almost no difference in write throughput.

Increasing the RDS instance class yielded a very modest gain, to maybe 1500 writes per second, but still far short of our goal and far short of the claimed maximum throughput of 3000 IOPS for the server configuration we were using.

😰

Round 2: Attempt to Isolate the Problem

The first round of tests were done in the full context of our application. We had instrumentation that was telling us that COMMIT was taking a while and, once we configured Amazon Performance Insights, we could see that a lot of time was spent waiting on a thing called object .

So what was this object thing?

We eventually found a description of it buried in the Postgres documentation for wait_event :

object : Waiting to acquire a lock on a non-relation database object.

Oh.

Not exactly helpful. So, really, we were still guessing.

False Alarm 1: The RDS service claims a particular throughput. One of our theories at this point was that Amazon was using this lock as a way to degrade performance to match this contracted level of service. The flaw in that theory was that the performance we were observing was actually well below the contracted level.

It was at this point that we decided to start fresh, building a new Elixir application and we would layer on the complexity bit by bit to identify the bottleneck.

False Alarm 2: At this point, I began to consider the possibility that Postgres itself was just slow.

To test this theory, I decided to boil the benchmarks we were running down to the simplest possible version I could conceive. There were two tests: one that created rows in a very simple table, and another that updated single rows in that same table at random. (Our application, at its core, has a fairly simple CRUD workflow.)

I then wrote a very quick-n-dirty application I named pghr (short for Postgres Hot Rod) ran these tests in a micro-benchmark using Elixir, Ecto, and the Benchee benchmarking library.

I shared the first results from that test with some teammates with the pithy comment:

Well, 💩. Back to basics and PG (or the way I’m using it) is failing a very obvious, simple test. Updates WAY slower than creates ?!?

(A footnote here: Elsewhere in this article and the pghr repo, I use the term “item.” The screen shot above is from an earlier internal version of pghr and it’s using a concept we use internally called pool . I don’t have a corresponding screen shot at this stage.)

A teammate had been experimenting with running similar tests using pgbench , the canonical load-generating / benchmarking tool for Postgres, and his results were far better than mine.

False Alarm 3: I now began to consider the possibility that Elixir and Ecto were just slow. I think it’s important to label this as a false alarm now because it was later decisively disproven, but at the time, it was a possible interpretation of the facts on the ground as we saw them.

Round 3: Ask for Community Help

At this point, having no obvious way to remedy the problem, I re-created pghr as a public repo, shared my findings in a post on Elixir Forum, and asked the community for help.

Finding 1: Choose Tooling That Is Appropriate for the Job

The first piece of actionable feedback was that my use of Benchee was ill-advised. Two major concerns were raised:

I was misinterpreting the output from Benchee when using parallel . When using parallel , Benchee does in fact spin up multiple Elixir processes, but only reports the throughput from a single process. So, net throughput was actually much better than I had understood. Thanks to Ben Wilson for stating that clearly. Benchee’s design space is really about microbenchmarks for individual functions. Running parallelized load-generation tests is outside its intended use case. Thanks to OvermindDL1 and PragTob, Benchee’s author, for stating that clearly.

So, I rewrote the Elixir version of pghr without Benchee.

It’s important to note that this is not a critique of Benchee. I continue to use this library in other contexts; I’m just now better informed about its intended design space.

Results for the create items benchmark were much improved and at least within the same order of magnitude as pgbench .

Results for the update items benchmark remained awful. pgbench appeared to be beating the Elixir code by a factor of 15 or more.

Finding 2: Enum.random Considered Harmful

I then raised this discrepancy between create and update performance as an issue in the Ecto project. José patiently worked through the issue and found the next culprit: Enum.random/1 is linear on the number of items in the list. As I was pushing the size of the list of IDs larger and larger, this became a substantial bottleneck.

This would be an embarrassing thing to share, except for this little tidbit that José posted on Elixir Forum a bit later:

Btw, thanks for putting the benchmarks up, they led to a batch of improvements on the Enum.random implementation in Elixir master!

Finding 3: Elixir and Postgres Are Actually Very Fast

Having rewritten the update benchmark to remove the use of Enum.random , we were now starting to approach the performance observed with pgbench . I made the following tweaks, based on suggestions from the forum:

Use cached statements. (#13)

Use Repo.checkout to reuse connections rather than checking them out for each iteration. (#14 and #20)

to reuse connections rather than checking them out for each iteration. (#14 and #20) Remove string interpolation from inner loop. (#15)

Add a warm-up phase to avoid code-loading issues. (#19)

Remove random number generation from inner loop. (#21)

After making these changes, I was able to show pghr performance to be between 87% and 107% of the write performance observed by pgbench . (Yes, in some cases, Elixir and Ecto were faster than pgbench !) This result was shown on two different laptops that I use. Similar results were demonstrated by others, using different hardware and OS configurations, on Elixir Forum as well.

Finding 4: pg_notify Considered Harmful

I said earlier that we were facing a major performance cliff in our internal application.

The original purpose of pghr was to replicate pgbench -ish performance with the simplest possible emulation of our application and then layer in complexity until we found the performance bottleneck.

Having now achieved that level of performance, it was time to start layering in the complexity again.

We had a suspect already in mind and it turned out to be correct.

We had built a trigger that called pg_notify ; that output was then consumed by our Elixir application as part of a long-poll response mechanism.

I re-introduced the pg_notify call into the schema in pghr . Just adding it to the database schema, without any listeners, severely affected performance.

As Ben Wilson stated on the forum, pg_notify uses an O(n²) algorithm to deduplicate notification. When you place Postgres under enough load, that becomes a bottleneck.

Unfortuantely there appears to be no mechanism for disabling that de-duplication, so the remaining answer is to re-architect without pg_notify . We’ve been able to take that back to our internal application's architecture, and we are now near our desired perf targets.

Finding 5: Elixir Community Considered Very Helpful

I can’t say enough to thank the community for their engagement through this saga. Special thanks to: