The great pretender: faster application tests with MySQL simulation
By: Matt Brown
Tags: MySQL, PHP, Testing

If you have a web-based application, chances are high that it uses some sort of database to store information. Chances are also high that you’ll want to test that application somehow. One of the questions you then have to answer is, “How do I test something that queries a database?”

Option 1: Limit database interaction to a mockable API

A tempting first step is to mock out high-level API calls. If you have a getCurrentUser method, a mock can ensure that method returns an object with appropriate data.

This is fine for some simple CRUD applications, but becomes an issue when you want to use MySQL features beyond simple SELECT <column> FROM <table> WHERE name = <value> queries.

Option 2: Mock out database calls directly

You could have something like this:

function testHighScores() {

$db_connection = new MockPDO();

$db_connection->receiveQuery(

'SELECT `name`, MAX('score') as `high_score`

FROM `game_scores`

GROUP BY `name

ORDER BY `high_score` DESC'

)->andReturn([

['name' => 'Kathleen', 'score' => 6700],

['name' => 'Will', 'score' => 6200],

['name' => 'Matt', 'score' => 2300],

]); assertNotEmpty(

(new ScoreRepository($db))->getHighScores()

);

}

This is fast, since your application doesn’t actually interact with a database during the test. Mocking everything also helps your tests conform to the platonic ideal of what a unit test should be — you can effectively isolate each component being tested.

But tedium can set in quickly: mocked query results become stale if the query changes, or when the underlying table schema changes. This causes maintenance headaches if you’re at a fast-growing company.

Option 3: Run tests against an actual database

You can avoid issues of staleness by running your tests against a real database:

function createScoresTable($db_connection) {

$db_connection->query(

'DROP TABLE IF EXISTS `game_scores`'

);

$db_connection->query(

'CREATE TABLE `game_scores` (

`id` INT(10) NOT NULL AUTO_INCREMENT,

`name` VARCHAR(16) NOT NULL DEFAULT '',

`score` INT(10) NOT NULL,

PRIMARY KEY (`id`)

) ENGINE=InnoDB'

);

$db_connection->query(

'INSERT INTO `game_scores` (`name`, `score`)

VALUES ("Matt", 20), ("Matt, 1200),

("Matt", 2300), ("Kathleen", 6700),

("Will", 6200), ("Will", 4800)'

);

} function testHighScores() {

$db_connection = new PDO(getConnectionString());



createScoresTable($db_connection); // do some sort of test

assertNotEmpty(

(new ScoreRepository($db))->getHighScores()

);



// clean up after test

$db_connection->query('TRUNCATE `game_scores`);

}

That introduces some new challenges — you have to figure out how to replicate your database schema, how to generate test data, and how to scrub the database once a test is complete, but otherwise it’s pretty reliable.

For over a decade at Vimeo most of our PHP tests have required a database connection. Though this has avoided some of the issues around mocking, it’s also pretty slow. Over the years we’ve come up with a number of different ways to make tests faster, using in-memory storage, collocating test databases and caching test results as much as possible, but a full test run still spends 50 percent of its total runtime talking to MySQL.

Why is using a real database so slow?

Setting up a database connection and communicating with an external service involves a lot of digital bureaucracy. This is especially noticeable when running tests in a virtualized environment (like inside Docker for Mac).

Databases are normally designed to hold a lot of information, and to hold onto that information for a long time. But when testing we usually want to wipe the database between each test, and tests normally write a tiny fraction of the data the actual production database would hold.

This slowness has led us to explore a fourth option.

Option 4: Simulate MySQL itself

What if we could use a library (written in the same language as our tests) to parse and execute MySQL queries — mimicking the behaviour of a real database? That library could be part of our test runtime, avoiding any interprocess communication.

function testHighScores() {

$db_connection = new FakePdo(getConnectionString());



// create tables with SQL queries as above

createScoresTable($db_connection); // do some sort of test

assertNotEmpty(

(new ScoreRepository($db))->getHighScores()

);



// clean up after test (as you would with a real database)

$db_connection->query('TRUNCATE `game_scores`);

}

Two months ago that library didn’t exist, at least not for PHP. Luckily there was something very similar — Slack has created their own MySQL engine, written in a PHP-adjacent language, to speed up their unit tests.

Slack’s testing story is slightly different from Vimeo’s: initially they chose to mock out all database query results when running tests. In 2015 they started simulating some simple SELECT / UPDATE queries, but more complex syntax (such as queries with joins) still required mocking

In 2017 the maintenance burden of mocking those queries became so costly that Slack decided to go all-in on MySQL simulation, and a developer there created an engine that supports all the MySQL queries they throw at it.

Hack, the language that engine is written in, is similar enough to PHP that it’s possible to roughly convert the code to PHP — I built a toy transpiler for that purpose two years ago. Over the holidays I used that transpiler on Slack’s MySQL engine, adapted it to play nicely with PHP, and added a number of MySQL features (like variables and auto-incrementing indexes) that we use at Vimeo.

That library, the boringly named PHP MySQL Engine, is now open-source. Ninety-nine percent of our tests use it in place of an actual database. During a full test run it processes everything from a simple SELECT `id` FROM `foo` to a 1,100-line behemoth that’s basically the final boss of MySQL parsing and execution.

Caveats

Whenever you’re simulating anything for tests, you run the risk that your simulation is slightly off, leading to tests passing that really should fail. It’s also very difficult to replicate MySQL exactly (one example: MySQL’s inference for computed column types is occasionally a bit wrong).

There are also many MySQL features the PHP engine does not support: anything that relies on stored procedures, triggers, or views cannot be simulated by it.

Anyone who’s decided to venture down the route of simulating MySQL should make sure they’ve first thoroughly evaluated mocking, and running tests against a real MySQL database. Read this article if you want to get cold feet.

Benefits

It’s much quicker — in CI, our unit test runs are on average two times faster than when using a real MySQL database.

Even better, it’s 4–5 times faster for our developers on Macs who previously had to run all PHPUnit tests under Docker, as our pure-PHP test environment no longer needs any virtualization.

If you’re wondering how PHP could be faster than a MySQL (a highly-optimised C binary) it’s all about the overhead connecting to a MySQL server, sending the data and waiting for a response — and doing that hundreds of thousands of times over a full test run.

My hope is that this performance boost will make testing a bit more fun.

Where to next?

Our work with PHP MySQL Engine is basically finished, at least for now. It’s not perfect, but it does exactly what we need it to do.

Aside from speeding up tests, there are a few other things this engine could be used for in the future: