Detecting and fixing crashes
By: Erica Riello
Tags: Women In Tech, Diversity, App Development, Tech, Experiment

No matter how much time Tech and Product professionals spend planning a new feature, its development is likely to introduce new bugs to the application. It can happen due to user unpredicted behavior, bugs on dependencies or services temporary unavailability.

OLX Brazil has around 150 people prototyping and developing new features for mobile apps and website. In this scene, we are all going to deploy buggy code once in a while. We have struggled to create an accountability culture in which everyone monitors app releases and website deployments.

Expecting people to actively look for evidence that their work is failing while they are producing the next piece of work is not only unsettling but also kills productivity. Furthermore, teams are often focused on addressing urgent services availability, scalability or responsiveness issues. These setbacks combined with pressure to release new features can lead to a very stressful routine for software engineers.

The ideal software development process is to have everything automatically tested. Even though automated testing is an amazing feature we can rely on, sometimes it’s hard to enumerate all use cases, especially when we are dealing with a large number of overlapping experiments concurrently running on the same page or screen.

In OLX Brazil we are successfully building an experimentation culture: In 2018 OLX had 132 experiments running in 4 platforms; in 2019, we ran 407 experiments. Most features released in 2019 had its value proven by metrics obtained while running an experiment through OLX Experimentation platform. During that year, we had some great new features whose rollout had been delayed because of app crashes found in experiments (thanks to Crashlytics).

Image created by freepik

Via the Experimentation dashboard, it is painless to change an experiment configuration. It takes approximately 15 minutes and does not require a new deploy or app release. One very useful configuration is to control the number of users segmented. By doing this, we can progressively release a new mobile app feature by scaling up the experiment’s population. This seemed like a great idea because, if the new feature introduced new crashes, we would simply turn the experiment off.

Monitoring Crashlytics while scaling user access to the new feature became the established procedure, but it was not very reliable since many other teams were also scaling their own experiments. Moreover we could spot crashes in the class or module used to trigger the experiment specific behavior with some ease, but finding crashes happening as a side effect of the new feature was still a toilsome task.

We tried using Crashlytics keys to list all experiments running in a user session when a crash happened. However, as the experiments grew, the list was long and Crashlytics interface did not provide an easy way to sample multiple crash occurrences, let alone verify if all occurrences had the same value for a key.

“Experimentation pipeline could help”, we thought. We have been calculating hundreds of app metrics every day for experiments, but none for crash free users percentage for each variant. We provide a graphic interface to compare variants results and tables where developers, analysts and product managers can explore and find patterns. It would take a look on our dashboard or querying experimentation data to spot crashing experiments.

We implemented a small piece of code to emit crash events in our Android app so that our experimentation platform could calculate the percentage of users experiencing crashes in the app. That 60 line Kotlin class was deployed and 2 weeks later two teams were digging their code into those small bugs that explained the 5% difference in crashers metric between their variants.

So now teams could easily determine whether their features code were safe to be released to all users, but if there was relevant unconformity of crash metrics between variants, developers were having a hard time searching for the unknown crashes. As early stated, exploring Crashlytics was a longstanding process that hardly succeeded.

We decided to enrich crash events with data about the thrown exception such as error message and stacktrace. First, we stumbled upon a problem regarding Out Of Memory (OOM) errors. Most of the times OOM errors happened, the app didn’t have enough memory to send an onerous event, which led us to the classic ‘OOM while trying to throw OOM’. We decided to omit the stack trace of such events, replacing it with a flag indicating OOM crash.

It’s still tricky recognizing similar crashes occurring in different operating system versions because they produce slightly different stacktraces, but developers can point out the exact line the app is crashing. Adding the stacktrace to crash events revealed corner cases such as experiments that did not introduce new crashes but led users to faulty flows that increased the app chances of crash by 14%.

We almost ended there: experiments are in constant development until proven crash free so we were not deploying more faulty code and the app was healthier, but we could do better. Until then, experimentation was used to avoid new crashes but now we’re using experimentation to solve long forgotten crashes.

Our app had many intermittent crashes we were unable to reproduce, but one of them affected almost 12% of monthly crashing users. A first fix was pushed before the crash event feature was released, but it seemed to have no effect on this particular crash metrics according to Crashlytics. We developed a second fix as an AB Test and deployed it. After running the AB for 6 hours we proved the second fix corrected the faulty behavior and did not introduce new crashes.

Chart of number of users affected by a single crash through time

This chart shows the number of users affected by the aforementioned between January 10th and February 9th. On January the 8th we released an app version with an experiment fixing that crash. Orange bars shows the number of users experiencing crashes in earlier versions, whilst purple bars shows the number of such users using the version in which we introduced the experiment fixing the crash. The experiment was turned on January 13th. After 6 hours we confirmed the fixed variant had significantly lowered the crash free metric and started scaling up the experiment to all users, fastly decreasing the number of users affected by this particular crash.

We also created a bot for those experiments metrics that send Slack messages for product teams when an experiment is raising the crashing users rate metric. Up to this point, developers and product managers had to actively search for the metric in Experimentation Dashboard and this requirement often delayed imperative interventions. After the bot started alerting developers about these undesired side effects, we intervened faster and delivered more stable features to our end users.

This bot runs daily checking app experiments for extraneous variation of number of users whose devices crashed and notify the owner team which variants are problematic and the effect of that variant as shown on the message below*:

Sample of Slack message sent to teams whose experiments in which users are experiencing more crashes than usual.

These messages are also sent to a second channel, created to centralize those alerts, in which members can be notified of any experiments causing crashes on the app. At first it can seem a tool for blaming teams for crashing the app, but the aim of this channel is to enable the whole enterprise to understand how new features have an effect on the apps stability.

These features combined with other initiatives developed by product and technology teams are supporting the construction of an accountability culture based on collaboration, autonomy, transparency and data.

We are tremendously excited about our next steps, that include releasing a similar iOS feature and enhancing the app stability by decreasing the amount of Out Of Memory errors crashing the apps.

* Data generated for this image is random, does not reflect any of OLX Brazil’s experiments configuration, it’s results or team names’.