Success case: follow the goal, not the requirement
By: Mirco Porcari
Tags: Outcome Over Output, Ratings And Reviews, Scrum, Product Ownership, Okr

Several months ago we introduced OKR in our scrum-based process.

We build a personal cloud product made of a backend, an Android and an iOS app, a web portal and apps running on computers. Our business model is B2B2C, meaning that we sell the product to big companies, usually mobile operators, who integrate it into their systems, rebrand the apps and make them available to the end users.

Among the first set of Key Results associated with the Objective, we had to improve the rating in the app store for our iOS app.

A good rating is very important for several reasons directly tied to the business: the app becomes more visible, the rate of users installing the app improves, the activations increase, and so on.

One way to achieve this goal would be to work on the app quality, making it nicer and fixing some bugs, and indeed we did some improvements in these areas as well. However, we mostly focused on the rating process itself, which had to change also for compliance with some recent Apple guidelines.

The requirement

A comparison analysis with other apps was performed, and the input coming to the scrum team was to introduce a new interaction layer before the actual invitation-to-rate popup. In that layer, unhappy users would be invited to send a direct feedback, without going to the store. If this behaved as expected, happy users would give to the app 4 or 5 stars, thereby improving the rating.

In other times, the team would have worked as a feature factory, with the goal to build the requirement in a stable and beautiful way, as fast as possible.

This time, having a clear and explicit KR made the difference: the goal, or the expected outcome using an established terminology, was to increase the rating. In a problem solving mindset, the requirement was seen as a suggestion to get there, with the freedom to consider other paths. Definitely this change of approach brought more energy and engagement in the team.

Towards the sprint goal

During the sprint, UX and DEV people together actually worked on changing the flow (let’s call it the WHAT), introducing the ‘send feedback’ as required. While digging into the existing code, with the goal in mind, they came up proposing an orthogonal approach: why not also focus on the trigger that makes the flow show up in the user’s path (let’s call it the WHEN)?

In particular, the existing flow was presented after the successful completion of n actions in a row, counting any action. The team suggested instead to identify a few emotionally valuable actions, and present the flow upon the execution of one of those: it was likely that in such a moment the user was more willing to provide a good rating.

Going live

Once the changes were ready for being launched to get the first empirical results, we faced a showstopper: for bureaucratic reasons, our customers could not deploy the ‘send feedback’ as it was designed, a more complex implementation was needed.

We then decided to just quickly disable the WHAT, restoring the flow to how it was before, and send live the app with only the change to the WHEN as proposed by the team.

The results exceeded our expectations by a large margin. Here is the trend of the app store ratings for two of our main customers, monitored over a period of a few weeks:

App store ratings for the iOS app, labeled and branded by two different customers

Such a rapid improvement is the result of not only better ratings by each user, but also of a much higher number of users rating the app.

What’s next?

Having seen things going so well, we decided not to invest any more effort on the rating mechanism, except for applying the new logic to our Android app, which also got a boost in the Play store.

As of now, we keep monitoring the apps in the stores, and evaluating sprint by sprint if any of the new features that we introduce should be added to the list of triggers for the app rating invitation to appear.

Conclusions

As a result of this activity we could observe that keeping the team involved with the current company Objective and the related Key Results brings more motivation, better team work, and a higher quality in the decisions, as everyone can contribute with different perspectives; the price to pay, which — as a Product Owner — is to be able at a certain point to take a decision of what to do and, much more important, what not to do (by now), seems worth the benefit.

In this particular case, without the team’s input, the activity would have resulted in a complete waste of effort and time.