Android A/B testing Made Simple With Cadabra Library
By: Dmitry Si
Tags: Development, Mobile App Development, Kotlin, Android, A B Testing

A/B testing is not easy: designing an experiment and analyzing data properly is a massive task, and technical implementation adds even more complexity to the mix. In this article, I will show how to streamline that last part of the A/B testing process on Android to focus on what matters the most — your data.

A/B testing complexity

Let’s start with the complexity associated with A/B testing before I offer the solution. So, what are the problems?

Async communications — the configuration of the experiment is often located at the server so the client needs to fetch it first, and this operation is asynchronous. That brings many questions: how do we show the UI? Do we need to wait? How do we check the latest configuration is loaded?

— the configuration of the experiment is often located at the server so the client needs to fetch it first, and this operation is asynchronous. That brings many questions: how do we show the UI? Do we need to wait? How do we check the latest configuration is loaded? Multi-layer changes — with most modern apps adopting one or another form of layered architecture an A/B test touches almost every layer in the app: it needs to be received from the network, stored somewhere in persisted storage, tracked on the domain-logic layer, and displayed through UI layer. That’s too many places to make a mistake (e.g see this article).

— with most modern apps adopting one or another form of layered architecture an A/B test touches almost every layer in the app: it needs to be received from the network, stored somewhere in persisted storage, tracked on the domain-logic layer, and displayed through UI layer. That’s too many places to make a mistake (e.g see this article). Multi-experiment tracking — with more and more experiments added to the app it’s harder to track which experiments are present at all, which can be active at the time, and whether they interfere with each other.

— with more and more experiments added to the app it’s harder to track which experiments are present at all, which can be active at the time, and whether they interfere with each other. Sunsetting — after the experiment is over one needs to go and clean up the code from all the layers affected at the previous step. Things like configuration models and resources often get forgotten along the way and stay in the code base for months after they aren’t needed anymore.

— after the experiment is over one needs to go and clean up the code from all the layers affected at the previous step. Things like configuration models and resources often get forgotten along the way and stay in the code base for months after they aren’t needed anymore. Boilerplate — UI changes may be as simple as a single button’s color change or as complex as a whole new layout, in both cases at least one if-else/switch-case is required. And all should be done in a way they won’t get lost during sunsetting.

Solution requirements

If we were given enough time to design a complete solution to the aforementioned problems how would it look like? Here is my proposed library requirements. (Scroll to the last paragraph to get straight to the code samples)

Fully synchronous experiment configuration fetching — most of the tools that define the cohorts for A/B not making the decision in real-time, and not delivering it instantaneously either, there is no point in trying to get the current state of the server-side config: whatever state was actual at the moment of app launch should be the state of the A/B test deeper in the app flow (BTW recommended config fetching interval for Firebase is 12h). That doesn’t just reduce the bandwidth usage and increases responsiveness, it eliminates tricky bugs caused by fetching the updated config while one or more experiments have already been shown to a user.

— most of the tools that define the cohorts for A/B not making the decision in real-time, and not delivering it instantaneously either, there is no point in trying to get the current state of the server-side config: whatever state was actual at the moment of app launch should be the state of the A/B test deeper in the app flow (BTW recommended config fetching interval for Firebase is 12h). That doesn’t just reduce the bandwidth usage and increases responsiveness, it eliminates tricky bugs caused by fetching the updated config while one or more experiments have already been shown to a user. Ability to enable the experiment locally — A and B groups for an experiment should be defined in advance and stay intact (that’s the best way to keep them truly independently randomized) but the experiment itself may not be available yet to a user until a certain event happens. It’s then required to activate experimental UI A or B immediately after, ideally without network round-trip, or extra code that checks for experiment activation status.

— A and B groups for an experiment should be defined in advance and stay intact (that’s the best way to keep them truly independently randomized) but the experiment itself may not be available yet to a user until a certain event happens. It’s then required to activate experimental UI A or B immediately after, ideally without network round-trip, or extra code that checks for experiment activation status. Single point of configuration — to keep track of all experiments we need to keep them registered in one place so it’s easy to see what’s active now. We’ll also need this point to set the configuration for integration and functional tests.

— to keep track of all experiments we need to keep them registered in one place so it’s easy to see what’s active now. We’ll also need this point to set the configuration for integration and functional tests. Statically typed enum-like configuration —once the experiment is started the sub-parameter can not change otherwise we’ll get inconsistent data. That immutability, in turn, allows us to create a data-class with a complete set of required parameters for each experiment variant. The enum nature will prevent us from accidentally skipping one of the options within the switch or when statement. Once the experiment is over we can remove the class, and the code won’t compile until we properly remove all the places the experiment was affecting.

—once the experiment is started the sub-parameter can not change otherwise we’ll get inconsistent data. That immutability, in turn, allows us to create a data-class with a complete set of required parameters for each experiment variant. The enum nature will prevent us from accidentally skipping one of the options within the switch or when statement. Once the experiment is over we can remove the class, and the code won’t compile until we properly remove all the places the experiment was affecting. Automatic resources resolving — many experiments are as simple as showing a new layout for the same data. It would be much more convenient to provide two layouts like cart_screen_variant_compact / cart_screen_variant_verbose for an experiment that has variants “Compact” and “Verbose” and let the framework inflate proper layout automatically. And during the sunset phase, we can then remove all the resources with either _verbose or _compact suffix and be sure nothing unused is left.

— many experiments are as simple as showing a new layout for the same data. It would be much more convenient to provide two layouts like / for an experiment that has variants “Compact” and “Verbose” and let the framework inflate proper layout automatically. And during the sunset phase, we can then remove all the resources with either or suffix and be sure nothing unused is left. Testability — we don’t want to see a random experiment’s variant during the testing phase, so there should be an easy way to launch any given variant, and it should be possible to mock the experimentation framework when needed.

Solution

With all these requirements in mind, I’ve created a library let’s see how that works (links to the library repo and maven below)!

Here is an example of the minimal A/B test configuration, for a simple test.

Both layouts and content can be changed by Cadabra

(I’ll cover more use-cases in a separate article, but the repo already contains the sample project, and the documentation describes the solutions for the most common cases)