How we’ve improved our AppStore rating with Firebase A/B Test and Remote Config
Tags: Firebase, A B Testing, Firebase Remote Config, Mobile App Development, App Store Optimization

As I explained in a previous post, Gympass App in 2018 was a full-screen WebView, with a poor User Experience, slow rendered and no usage metrics. No surprise that our Play Store rating was 2.3 and the AppStore below 2.0. This means our users hated Gympass? No! Actually, no. Our users love our product, but they hated our platform.

Nobody gives a 1 star rating to Facebook because they don’t like it’s content, people are sending a message that there is something wrong happening (with the platform or product support, in our case). Let’s be honest, you may have one or other user rating 1 star because they didn’t like your product, but not a thousand ratings like this.

Well, we’ve already had 1k ratings with 1 star! That used to represent 90% of our ratings.

However, now that we’re moving forward to develop a new experience, and the main features are already live (many enrolled users only use Gympass app to login and check-in) it’s time to ask those users to rate and review our app new experience!

Our check-in process is like publishing a content on instagram: the final result is a post to your followers, but there are many ways of building this content. Some users will see the final result of the post in their own timeline, other users won’t see it anymore. The same happens to check-in: some users will have to wait for the gym to confirm (countdown flow), and many of them won't wait with the app opened (validated flow).

So if you had to ask for users to rate your app, in which of these steps would you do it? Is there a best time to ask it? I DON’T KNOW, but maybe I know how to find it!

By using Firebase Analytics to track our users behaviour and Firebase A/B Test to measure the goal for two or more different groups.

I've also explained in the previous article how we have used Firebase Remote Config to choose which experience our users are going to see. Now we’ve created a new flag for which step the app may ask users to rate. There were two possible options:

And of course another key for turning this feature on/off.

To complete the environment, we would like to ask for reviews only if the user has had one or more good experiences. Thus, we’ve created another Firebase User Property to store how many times a user have checked in so far.

Now it’s easy to build our A/B Test.

For example, I want to know if users are rating and reviewing my app in the complete flow more than in the countdown flow.

My control group had the remote config flag to “countdown” and the A group to “validated”.

The event goal was ‘app_rated’.

DONE. After 2 weeks running the experiment we found that users were much more willing to rate our app in the countdown flow, and so we've decided to go on with that.

Still, we’ve seen some countries rating a lot and other countries with no ratings at all.

WHY? WHY, WHY, WHY!

Although we were happy to do our first A/B test, we’ve ignored a very important check-in business rule: some countries don't see the “countdown flow”, consequently they would never rate the app.

HUM, SO?!

We've decided to change the value of our remote config User-rating-flow for those countries that had few reviews, and BANG!

It's raining 5 star ratings

Now, we are back to the game ;)

To sum up, the A/B test did work for us to decide which flow was performing better in countries where both scenarios existed. But as in our product some countries have only one of the scenarios, Remote Config could solve this problem.

In this example, let’s say that German users should always see the dialog at the validation step and all other users at countdown:

That's it. That’s how we achieve 4.7 stars on AppStore when adopting ReactNative!

Have you ever faced a problem like that? Please, clap your hands if you have liked it and follow Gympass on Medium!