Visual regression testing at Doctolib
By: Thomas Bentkowski
Tags: Visual Testing, Testing, Continuous Integration, Quality Assurance

“There’s no hiding from us now.”

Here at Doctolib we run visual regression tests on our CI to be sure that our front end is never unintentionally modified.

Visual regression testing? Eh… What do you mean, mate?

It’s pretty straightforward: behind each build of our application, there is an automated test which highlights any visual difference between screenshots of a candidate branch and screenshots of the master branch.

The differential approach of visual tests

And that’s not all, we also, of course, have unit and integration tests. Altogether, we have roughly 10,000 tests running in our CI pipeline.

Ok ok, I see you seem to like testing your application, but if you already have unit and integration tests, why bother visually testing your front-end? Don’t all the scenarios you’ve written as integration tests already prove that your application works?

Why do we have visual tests?

As you may very well know, not everything is perfect and visual testing has both pros and cons. We really do like having visual tests because it grants us better access to automation, however, it does come with drawbacks. In this section we will try to be as honest as possible regarding visual testing at Doctolib and the level of satisfaction we have about it.

Proactive monitoring of our front-stack

Owning our front stack and getting as close as possible to it.

By making sure we add snapshots to new views, we ensure that we do not unintentionally introduce unexpected visual elements. This is helpful in that it increases our team’s familiarity with the views we produce.

Reducing technical debt

This is rather self explanatory : “Untested Code Is Broken Code”.

Reducing time spent on testing

There are around 450 snapshots taken for our visual test suite. This means we visually test around 450 different views or view states. By creating an automated script to run these tasks, you can run your test suite faster and more safely.

For example, let’s imagine you’re working on a new feature that involves three different views. Normally, you would have to manually check each of the three views after your development to ensure that everything has been rendered as it should be.

Let’s say that checking a view takes you three minutes. With a visual test you’ve already saved nine minutes thanks to the snapshots, however, don’t forget:

Your eyes will never catch the pixel differences that might occur

A Product Owner/QA will very likely perform the same checks as you just did

It’s impossible to know whether your new features have had an unexpected effect on other views

Maintain a higher level of quality

Reaching pixel perfect interfaces

Thanks to our visual tests we are now able to detect almost 100% of the pixel differences we introduce and thus ensure that we ship stable views to production.

Visual tests are more simple than they were before

Visual testing has always been quite painful. With a classic end-to-end approach test, you would end up with a scenario like this one:

Trying to make visual test the old fashion way

With a visual difference tool, a simple three line test can do the same job:

Adding a visual test

But what’s even more interesting, is the fact that a visual test will test far more that just finding elements inside your DOM. It will also check things such as:

Color differences

Block positions

Correct viewport

…

Visual tests are not only easier to write but they also provide a better coverage of your UI.

CI integration is a must

It is also easy to use because it is fully integrated in our build process thanks to Argos-ci (see below):

Visual tests runs concurrently with unit and integration tests

No extra work is required from a developer to run them

Tooling

Argos-ci

As mentioned, at Doctolib we use a tool called Argos-ci (which I will refer to as Argos from now on). Argos is a tool made by two previous fullstack Doctolibers™ Olivier Tassinari and Greg Bergé, and its end goal is to ask you a simple question:

Would you like to accept the visual modifications you have introduced?

From there you have two options, either accept the differences or fix the problem, but before we discuss this process let’s take a look at the tool itself.

Argos usage

A visual test

Once this line has been added, Argos will automatically compare the two screenshots and report the differences after each build we trigger. Since the tool fully integrates into GitHub, it will directly alert you if any differences have been detected (see below).

Our build report on GitHub

When you review the Argos report, you can identify the differences and decide on your next step given the two options available:

Accept the differences that have been introduced by the candidate branch

or

Fix the problems that have been introduced

An Argos report header

If you approve the differences (e.g: say you made a user interface evolution to a page), Argos will update the build status directly on Github:

Doctolib build status report in GitHub

Once every test suite has passed, you know it is now safe to merge your code into the master branch.

ImageMagick, quick dive

Argos itself is based on ImageMagick, the go-to reference for image manipulation.

According to the ImageMagick website it is used:

“[…] to create, edit, compose, or convert bitmap images. It can read and write images in a variety of formats […] to resize, flip, mirror, rotate, distort, shear and transform images, adjust image colors, apply various special effects, or draw text, lines, polygons, ellipses and Bézier curves”.

What the website does not directly tell you is that it can also be used to detect differences between two images. The output of this comparison is a third image which highlights these distinctions, thanks to color replacement. If you want to learn more about this process you can find out more details here.