Jones Automation Journey
By: Or Kadouri
Tags: Puppeteer, Automation Testing, Nodejs, Jenkins, Cicd

I’d like to show you our automation infrastructure journey from the early days of Jones, all the way to an R&D department that is built on top of it.

Throughout my career, I often witnessed the pain of insufficient testing for software releases.

What does it usually look like?

Production bugs — affects customer satisfaction, adds customer support overhead, and disturbs engineering days (and nights).

Slow releases — the constant need to test everything from scratch on every change and this endless ping-pong game between developers and testers.

Budget impact — hiring more manual testers to bridge the gap between the existing testing level and the level we want to achieve.

There is a variance between how insufficient testing looks in different companies, but it always stems from the same issue — lack of automation.

Companies nowadays are aware of the issue and do their best to bridge the gap to improve their release cycles and overall cadence. Most of them are building automation teams from scratch and hiring more and more automation developers to cover years of development depth in this area. Unfortunately, many times, it is too little too late.

It is often counterintuitive to invest in automation.

How do you convince people inside an organization to invest in something that doesn’t produce immediate results or revenue? It is especially hard in a small startup that is low on resources and aggressively fights for every new customer.

Luckily, it was in our company’s DNA to do things right from the start. It was initiated by the founders and echoed throughout the rest of the company. So when we built our R&D department, we wanted to learn from experience, and do it differently.

Our first hire was an automation developer.

There is no better testimony of our R&D culture than the above sentence. We started building the product only after we had the infrastructure for its testing. All of us wanted to empower developers and make their lives easier with the thought that some investment, in the beginning, will definitely pay off later in our company’s future. The idea was that future development cycles (and releases) would be quicker because we already have a high percentage of testing coverage, so we know if something breaks and fix it immediately.

Phase 1 — choose the right technology stack

How did we start?

As a developer, the first thing you usually do is to choose the appropriate technology stack that will fit your needs for the next 2–3 years.

We wanted to develop automation quickly, after all, we were a very young startup.

Our long-term strategy was to treat automation as an integral part of every new feature development cycle, so we wanted tests to be created via coding.

The year was 2017, our idea of the product was that it would be web-based, we didn’t need to support mobile devices, and we are adventurous at heart, so we chose Puppeteer. The most serious alternative at the time was Selenium.

Puppeteer

For whoever is not familiar with Puppeteer, I have included the description from their official website.

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.

What can I do with it? Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:

Generate screenshots and PDFs of pages.

Automate form submission, UI testing, keyboard input, etc.

Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.

The main issue with Puppeteer is that it supports only Chrome (for now). We checked browsers’ usage statistics before the final decision, and Chrome controlled the market (with more than 60%), so we decided it was good enough in order to start quickly. TLDR: it worked. The number of issues we saw on other browsers after our tests passed on Chrome was minimal. After more than 4 years of using this library, we can say that the results don’t justify running all tests on other browsers consistently.

Phase 2 — a single test

We hired an automation developer, chose our working tool (Puppeteer), so what now?

In parallel to building the application, we started building the first automation test. The idea was to create a basic end-to-end test that will act as a “sanity” test to the entire flow. We called it in our build job (technologies: Jenkins, Jenkins Pipeline) that all of our application developers use to deploy their code to one of the environments. This way, the application developer will get very quick feedback if he broke something on the application main flow. The feedback loop was 3 minutes!

Phase 3 — multiple tests

Then it happened.

The application grew larger as more application developers joined our team. It got more complex as we needed to support more user flows. At the same time, the automation grew more complex as well. All of a sudden, a single test case wasn’t enough, so we created more. We got to 10 scripts and a running time of 20 minutes (feedback loop of 20 minutes vs. a 3-minute feedback loop before) until we stopped to regroup.

It was now time to use a better testing framework. We discovered Jest & Mocha and finally chose the latter. Quickly enough, the 10 scripts turned into 10 test cases built on top of Mocha. The final execution report was much better, our debugging experience improved, but the execution time was still 20 minutes.

Phase 4 — parallel tests

We transformed the scripts into Mocha tests, so everything was modular, but the feedback loop was long. As we hired more and more application developers, our ci/cd pipeline started to build a job queue. Everybody tried to get feedback from the automation before opening PRs or merging the code they worked on, but the automation infrastructure couldn’t support it. The problem wasn’t with the Jenkins machine (technology: EC2 instance). The issue was that our framework didn’t use the computing power the machine had to offer. It was about time to make the tests’ execution parallel. It was about time to introduce mocha-parallel-tests.

For whoever is not familiar with mocha-parallel-tests, I have added the description from their official website.

mocha-parallel-tests is a test runner for tests written with mocha testing framework which allows you to run them in parallel. mocha-parallel-tests executes each of your test files in a separate process while maintaining the output structure of mocha.

Note: currently, Mocha supports parallel execution in their core library, but a few years ago, you needed to use mocha-parallel-tests to gain this needed functionality.

Phase 5 — debug tests

It is very easy to debug when you have a single test or even multiple tests that are running sequentially. However, an environment with Mocha parallel is a multi-threaded execution environment, and multi-threaded development and debugging are usually much more complex.

In such a complex environment, it is usually very hard to understand whether automation failure is due to an application or an automation bug.

Is it an automation bug? e.g., some instability in the automation infrastructure

Is it an application bug? e.g., a UI module stopped working

We needed the tools to answer these questions, but we also needed to know where exactly the issue was (module/file/line of code).

So what did we do?

Logs : each test case writes its own logs. Each test’s logs are pushed separately to the output console so they won’t get mixed with logs from other tests. Each log line also includes the exact time it was written, this helped us detect problems that stemmed from parallel tests execution (e.g., 2 tests changed the same account) because we were able to determine on each specific time slot which tests were running.

: each test case writes its own logs. Each test’s logs are pushed separately to the output console so they won’t get mixed with logs from other tests. Each log line also includes the exact time it was written, this helped us detect problems that stemmed from parallel tests execution (e.g., 2 tests changed the same account) because we were able to determine on each specific time slot which tests were running. Screenshots : our automation is an E2E UI automation (reminder: Puppeteer). Often during debugging, we added code that would create a screenshot of the state of the browser during the failure. In the end, we just added it to the infrastructure — every time an exception is thrown, we automatically take a screenshot and save it to the cloud (for future offline analysis).

: our automation is an E2E UI automation (reminder: Puppeteer). Often during debugging, we added code that would create a screenshot of the state of the browser during the failure. In the end, we just added it to the infrastructure — every time an exception is thrown, we automatically take a screenshot and save it to the cloud (for future offline analysis). Network data : we saved the traffic between the browser and the application to show evidence of problematic network calls that caused the failure.

: we saved the traffic between the browser and the application to show evidence of problematic network calls that caused the failure. DOM state: when an exception was thrown, we also saved the existing Chrome tab’s DOM state to understand whether some selectors were missing or not. It helped us to find whether an issue was with the application or with the automation quickly.

Phase 6 — testing analytics

While Jones continued to grow, our automation team grew to 4 developers — and we had ~400 tests running in parallel. The feedback loop was around ~10 minutes, but this wasn’t the issue. Our issue was stability. On each automation run, 5–10 tests were failing randomly, and we weren’t sure exactly how to investigate it. Though it was only 1%-4% failures on every run — it is still time-consuming. We lacked visibility into what was happening. With this number of tests, we needed to be more methodic.

After some internal and external brainstorming (thanks to some good friends), we were able to construct our plan. First, we needed to add tracking to our tests infrastructure, monitor it for a while, and then act based on the concrete data that was gathered.

We created a tracking mechanism. The process is to read the ci/cd pipeline log after every job run (technologies: Jenkins, Jenkins Pipeline), analyze the successful & failed tests (Node JS), push the information to a DB (RDS, MySQL), and visualize it (AWS QuickSight).

The tools were chosen as we use most of them anyway for other purposes (it saved some time).

And it worked. We were able to find the tests with the lowest success rate easily and fix them quickly. Bringing all tests stability to 100% — this is an ongoing process. Most of our tests are already quite stable, and we are working on the rest.

Phase 7 — levels of testing

As we continued to grow, we started to emphasize different levels of testing during our ci/cd pipeline:

Step 1 — unit tests.

Step 2 — integration tests.

Step 3 — automation (E2E).

This helped us to find issues in branches/features even without running the full automation suite and saving a lot of wait time in the job queue. Unit tests + integration tests results are received within less than a minute, and in case they fail, it saves us from running the full 15-minute automation suite cycle.

However, we felt it was not enough. Often branches still had severe issues either in the interface between the backend and the frontend or in the DB. In such cases, all unit tests and integration tests passed, but the automation suite failed completely. It felt like running the full automation suite in such scenarios is a waste of time because 95% of the tests failed, so we wanted something that would catch it without waiting for 15 minutes.

To solve it, we split the automation suite into 2 types of tests:

Sanity test cases.

Regression test cases.

We made sure a sanity test length was no longer than one minute, so the feedback loop, in case of a severe bug, is very quick. Immediately after the sanity tests finish running, the execution of the regression tests will start.

And the final steps are as follows:

Step 1 — unit tests.

Step 2 — integration tests.

Step 3 — automation (E2E) — sanity test cases.

Step 4 — automation (E2E) — regression test cases.

Phase 8 — optimizations

Along the way, we made many other improvements, among them:

An option to run partial automation (only specific test cases) — to detect issues quickly.

Common configurations between test cases — allow us to create test cases very quickly based on existing data.

Tests generator — create a test from a template. It allows us to create all the boiler plate code needed to generate a test case automatically.

The future

Jones is growing fast, and our entire R&D department is built on top of our automation. As a result, this infrastructure needs to grow as well.

Some of the things we are working on:

Load testing — as the number of users is constantly increasing.

Dynamic environments support — independent spawning containers executing on-demand automation jobs in parallel to reach a minimal feedback loop.

Mobile application coverage.

Common automation infrastructure to replicate the success for more applications.

And so much more…

Want to join us?

Please check our open positions at

https://getjones.com/careers/

Why join Jones?