Make your UI tests green again
By: Andrei Mamchenko
Tags: Swift, iOS, Bux, Stub, Testing

Know your tools

I’ve never done any UI testing before joining BUX. In my previous companies, it was always a nice to have thing and it never happened that we were able to reach the state when you can do something that is not mandatory. Basically, it often happens to the most engineers in early-stage startups. I was really happy to see UI tests in our BUX code and moreover, the guys managed to build some nice abstractions over the basic Apple tools. I was happy to catch the ball and take an active part in the process of improving the quality of our application.

I will not try to convince you to write tests. You just should do this on a daily basis. It is the same question as “Should we use a version control system?”. The answer is “Yes, we should”.

If you still need some inspiration, you can start with the article written by a colleague of mine. You will find some awesome historical insights there besides the main point.

The main purpose of UI tests is to make a quick run of the most important scenarios. We should be able to track if we broke something from the user’s perspective. Definitely, we should see our tests failing from time to time! This is how tests help us improve the quality of our software. However, there are some cases when we can get red tests and the reasons behind seems to make no sense for us as we can not do anything to fix it.

In this second part, I would like to talk about the main reason that can make your tests red: networking.

Reasons to bother about networking

Imagine you’ve just finished writing your brand new UI test for the new part of the app’s flow. You hit Cmd-U, slightly smiling when you see that Xcode’s “Tests succeeded” and then commit. You feel everything is accomplished. But when it goes to CI, things might change. You will definitely see red tests one day. That may happen for many reasons:

CI environment is out of sync with the development environment (e.g. Xcode needs to be updated)

certificates are expired

network/backend failed (the most often and unwanted case)

Backend usually has a variety of reasons to not respond at all or to respond in an unexpected way:

server maintenance

network reachability issues (if you do testing using real devices and there is a weak WiFi signal in that box)

data related issues. We can’t always know what data we are going to receive from the backend

If anybody in your team asks you to use the real server for your UI tests, just say no. If they want to test the API without putting additional efforts into it, they should use some proper integration tests, not mobile apps. Using mobile UI tests as integration tests is not the best way to go, as we introduce several points of failure and this makes tests flaky and useless both for mobile and backend sides.

Even if an app can reach its backend server all the time (let’s assume the backend is always running), it is still a huge dependency we have no influence on! So every time we see red tests on a CI we should be sure that we broke our tests with a new commit. We aren’t going to check for backend/network issues every time. At the end of the day, everybody can get used to “red” tests and that is not the way we want to go.

The way to solve it

How can we overcome that? We should start by not using the real backend for our UI tests.

We decided to start with OHHTTPStubs as we have already done some stubbing using it. This thing is amazing and it saved us a lot of time. At BUX, we decided to go a bit further and wrapped this library to make things more clean and maintainable.

Our goal was to be able to create new UI tests easily without any struggles when it comes to deciding “Do I have time to write a test for this feature or…?”

Requirements

Our main requirements for the stubbing tool are:

stub files (a lot of them, currently we have ~300) should remain easily maintainable

the system should have a possibility to return different stubs for the same endpoint

we should be able to modify it to meet new requirements in a predictable and clean way

single test flow should be able to use different stubs with a specific order

There are some libraries available on GitHub and we’ve looked into it as nobody wants to reinvent the wheel. After some investigation, it turned out that mostly they cover some parts but none of them do all that we need. We decided to go on with our own tool.

To be short, that’s one of our UI test classes. Easy to read, easy to make changes to. OnboardingNinPage here is one of these Page Object Models (POM) described in the previous article.

We use base BXTestCase that provides a mechanism to pass some data into the app using launch arguments so the main application would know which stubs should be applied for the test run. Then we just override stubs configuration needed for specific tests in each test class. So the final hierarchy looks like BXTestCase -> OnboardingTestsBase -> OnboardingNonOptionalNinTests .

Here comes the stubber itself. I’ve listed our requirements, here it is how we approached them.

Maintainability

We need to store stubs somewhere and finally, they should land in the app’s bundle to be easily accessed from runtime. We’ve ended up with a simple Xcode Build phases script:

To avoid a mess in our stub files we’ve organized everything in folders that reflect API calls in a simple way.

So, the call to /core/3/users/12345/portfolio will be reflected in the folders structure pretty much the same way with only one change — we use wildcards for IDs. It becomes Stubs/core/_any_/users/_any_/portfolio. As a result, it is easy to navigate, modify and even copy paste to create new stubs.

And finally, the file itself contains a raw JSON response.

For the above-mentioned example, we use two stubs configurations: NinToPersonalDetails and NinAsCurrentStep . Behind the scenes, there is one more configuration named Default but it’s up to you to decide whether to apply all configurations explicitly or to have one applied by default. Here’s how the folder structure looks like:

NinToPersonalDetails

NinAsCurrentStep

Then we just pass configurations names to the stubber using launch arguments. The stubber reads all of them recursively from configuration folder(s) building stubs endpoints from folders hierarchy, wildcards and some metadata from file names.

Every time OHHTTPStubs intercepts network request, the stubber searches through in-memory stubs array trying to find the right one. If none were found, we crash the app with fatalError to make sure any network calls reached the real backend. Seriously, we don’t need this in tests.

Flexibility

Currently, in our REST API, we have a few endpoints that are supposed to return different responses for the same requests. One of them just returns the current state of the user’s onboarding process. It ends with /applications/current and is always used without any parameters. Within one UI test, it should return different states when called several times in a row to test how the application performs handling of state changes.

We achieved this flexibility by adding some metadata to the stub filename. In general, there are three parts in the stub filename:

HTTP status code that would be applied when the stub fires

“response” part to make it more human-readable

metadata modifier that makes a stub or stubs configuration to be used by the stubber only once

So, for stub named 200_GET_response_runStubOnce.json we get the 200 status code and this stub will be removed after it was used. This adds a lot of flexibility and we can achieve it easily with combining small, simple and maintainable stub configurations together.

Easy to modify

To be honest, we had no need to modify this from the moment we built it. Once we started using a second backend service, we added a second instance of the stubber into the main app.

Usage

When we define stub configurations overriding stubConfigurations from BXTestCase, they are stacked within the stubber. If there is more than one stub for the same endpoint, the one passed last has the highest priority. Usually, we have some default configuration applied by default and there are several (no more than three) stubs applied to it, mostly with run once modifiers.

Finally,

what should we do now to write UI test for one new screen?

add accessibilityIdentifier s (described here) to UI elements

s (described here) to UI elements add POM wrapper for the screen to hide all XCUITest things inside (UI elements shortcuts, visibility and accessibility asserts, encapsulate complex actions to reach some tricky UI elements inside helpers)

things inside (UI elements shortcuts, visibility and accessibility asserts, encapsulate complex actions to reach some tricky UI elements inside helpers) add stubs configuration(s) (could be done manually or copy-pasting the similar ones or using some automatic response writer)

create UI tests class, override stubConfigurations , write test logic

Sounds easy, doesn’t it?

Here you can look into the deeps: HTTPRequestStubber.swift

Appendix

Actually, we used some sort of response writer to create all stubs for the first time. The only thing that required some attention afterward was the process of replacing real IDs with wildcards to keep stubs clean and reliable.