Quality Engineering Transformation in Ralali.com
By: Irwan Rosyadi
Tags: Technology, Software Testing, Quality Assurance, Quality Engineer, Mocha

“It takes a month to find a customer, Seconds to lose one”

Hai! Ever think about maintaining customers by improving your product quality? Talking about quality, everyone definitely wants everything to come in perfection and fully function. Especially in technology, good software will create a mechanism where every new user will make its product more valuable to every other user (network effect) and more users will take more risk in our system if it occurs malfunction.

The Seattle-based coffee giant suffered a massive outage that shut down the point-of-sale systems at stores across the U.S. and Canada earlier today. The problem, caused by what Starbucks called a “failure during a daily system refresh,” occurred around 4 p.m. By our rough estimates — calculated by journalists, not accountants — that means Starbucks lost around $3 million on Friday because of system problems.

Source : geekwire.com

“So, will you still think that Quality Engineer responsibility isn’t important?”

Some of you should have heard about Software Quality Assurance and Quality Control. In fact we’ve been confused and equalize those processes into one, “they will do a couple of tests to assure products are deliverable or not”

In several other companies, they separate teams in Quality Engineering as Manual Test and Automation Team. But in Ralali.com, we call it Quality Engineer. In 2017 when I joined this company, we started with the manual test, followed by no documentation of test cases, no periodically regression test, and no monitoring platform to check whether the previous feature was blocked by the new one.

Once we’ve tried to implement our first UI Automation Test using Selenium by the end of 2017. Quite frustrating to maintain the success rate. It happens because of every new feature published or if there are enhancements in the design of our web platform, many of our tests come to fail and the success rate is going to fall. Our obstacle was there are no standards of element names that are used for our code. As a result, we start to abandon our UI Automation and go back to manual testing only.

Many UI framework or platforms that we’ve tried before, such Selenium, Ghost inspector, Cypress (it’s started to use now), and Katalon, but we are still struggling with the same problem: “UI Testing is Flaky”

Test Pyramid by Mike Cohn

After we learned from the Test Pyramid above, we know where we should start first. Unit-test code comes from our Backend Engineer, and the second layer is our responsibility as a Quality Engineer (API TEST).

Before we start to implement API testing, we have to strengthen our test plan documentation. It is mandatory to create test cases and update it along with the sprint. We use one of the Test Management Tools, Qase.io. After that phase, the journey starts here. Mocha & Chai was chosen as our API test automation framework.

Starting from our monolithic API, based on our test cases we start implementing our API testing and put CI/CD on it for our regression test. Day by day we keep increasing our coverage test and adding more cases that can potentially occur. After that, we keep our test to the microservice level and also implement CI/CD on it. At present, many of them have 100% coverage and a 100% success rate.

The benefits of API testing that we know in advance if there are errors on one of our API before it goes into our production phase and is released to the public.

Maintaining the API test success rate is more efficient than the UI Test because the API change happens rarely. API test success rate fluctuates, but we never know whether our success rate is increasing or decreasing. So, we decided to implement our Monitoring Dashboard (we use Grafana as our monitoring platform). Its function is to record the number of our failed cases, tested cases, and our test success rate. When the success rate suddenly decreases or the number of failed cases increases Grafana will send a notification to the Quality Engineer group channel and we will check the root cause of the incidents.

The Process How Grafana Dashboard Generated (Monitoring Dashboard)

Generated Grafana Dashboard (Monitoring Dashboard)

Alert to notify the failed test result

Thank you to my team for this journey, but we still have a long way to go to get our quality process being the better ones, and the battle still goes on.

Are you interested to fight on our side? Come and join us!