End-to-end testing is a testing methodology that checks if an application behaves properly from start to finish.  Those tests ensure that all the pieces of the application work together as expected.  One common misconception is that end to end tests are front-end tests.  While we mainly interact with our front-end, the test scenarios rely on our back-end, infrastructure, and database structure. 

So how did we end up here? 

Nightmares of release day

In my past experiences, we traditionally tested our releases manually.  Release days were dreaded by both engineers and testers.  We would come to work and test the latest release candidate by manually executing user scenarios. 

Those scenarios were regression tests to ensure the core features of the app still work:

Can the user log in? 

Can they invite other users? 

Can they buy stuff? 

And so on…

I have seen huge excel spreadsheets with 50 cases on each of them.  Manually verifying these features was tedious, and we had to repeat it for every build. 

If a bug was found it was sent back to the devs for a fix.  We would then perform yet another round of scenarios to rule out any side-effects of the fix.  Once the fix was merged we would start the tests from the beginning and would go through that process until we had a bug free release. 

At some point, sometimes very late at night (really early in the mornings), we would deploy our app to production and go home feeling accomplished but tired.  It was quite draining for the team, sometimes I felt like testing my sanity more than testing our software. 

Let’s automate it

At Intercloud we decided to do things differently.  We are a small team of 10 awesome people (Hiring btw 😊) and cannot afford to spend that much time testing manually.  That’s why we decided to make automation a priority and we chose Cypress to help us. 

Cypress is a modern web automation test framework designed to simplify browser testing — and it’s much more than just an end-to-end test automation tool.  We use it to interact with our front-end app built with React/Redux and Typescript. 

Here is an example of a simple test that checks if we can still create a user. 

Made with Carbon

Result:

Today, we have about 179 tests spanning across 16 test suites. 

Easy to set and configure Cypress quickly became one of the most starred GitHub repo of this year.  Thanks to a visually appealing way to run integration tests, we managed to quickly cover most of our codebase. 

For me, the killer feature is the tooling and the ease of use.  As a developer, I know that it’s very easy to skip on testing some cases due to how annoying and not fun it is.  Cypress made me fall in love with testing again. 

TL;DR: Cypress is awesome. 

More than just UI testing

The whole point of End-to-end testing is to test whether the flow of an application is performing as designed from start to finish. 

Thanks to Terraform, we can spin up a whole dedicated environment for the tests and keep it isometric to our production environment.  This is quite huge since we have a rather complex architecture consisting of distributed microservices built-in Go running on AWS services. 

Made with Cloudfraft

This is fun and modern and all but rather complex to test in its entirety.  We have discovered bugs related to connectivity, race conditions and infrastructure bugs. 

Add it to your CI/CD process

Automation doesn’t stop with user interaction.  In order to use end to end tests at their full potential, we integrated it into our development pipeline.  We use Gitlab CI schedules to run them every morning before we arrive at the office.  If something broke we will get a notification on Slack prompting us to investigate the issue. 

We also use cypress awesome tooling to provide us with a very simple and concise test report and screenshots when the test fails.  Cypress also allows for video recordings which could allow you to get a replay of scenarios that failed. 

The problem with automated UI tests is that they are slow as they need to interact with a browser, even when running in headless mode, as they need to send and receive data.  Hence, the feedback loop is slow.  Detecting a bug the next day is still way better than discovering on release day. 

Another issue with automated UI tests is that they are quite often brittle, meaning that they could fail for a variety of reasons, often due to valid failures, but sometimes due to false positives which makes it hard to get a Green build consistently. 

We are currently still working on those issues but are becoming very consistent with the results of our tests. 

Conclusion

End to end testing has been a fun challenge and positive addition to our workflow.  It added a boost of confidence in our codebase and our Infrastructure. 

One change that I noticed was a cultural one.  Instead of relying on a QA or a testing team I felt responsible and took ownership of the quality of the software that I delivered. 

Want to join us? 

By Georges Antonio Indrianjafy