Predicting Crashing Releases of Mobile Applications
Xin Xia1 , Emad Shihab2 , Yasutaka Kamei3 , David Lo4 , and Xinyu Wang1∗
3

1
Zhejiang University, Hangzhou, China; 2 Concordia University, Montreal, Canada
Kyushu University, Nishi-ku, Japan; 4 Singapore Management University, Singapore{xxia, wangxinyu}@zju.edu.cn, eshihab@cse.concordia.ca,
kamei@ait.kyushu-u.ac.jp, davidlo@smu.edu.sg

ABSTRACT
Context: The quality of mobile applications has a vital impact on
their user’s experience, ratings and ultimately overall success. Given the high competition in the mobile application market, i.e., many
mobile applications perform the same or similar functionality, usersof mobile apps tend to be less tolerant to quality issues.
Goal: Therefore, identifying these crashing releases early on sothat they can be avoided will help mobile app developers keep theiruser base and ensure the overall success of their apps.
Method: To help mobile developers, we use machine learningtechniques to effectively predict mobile app releases that are more
likely to cause crashes, i.e., crashing releases. To perform our prediction, we mine and use a number of factors about the mobile
releases, that are grouped into six unique dimensions: complexity, time, code, diffusion, commit, and text, and use a Naive Bayesclassified to perform our prediction.
Results: We perform an empirical study on 10 open source mobile
applications containing a total of 2,638 releases from the F-Droid
repository. On average, our approach can achieve F1 and AUC
scores that improve over a baseline (random) predictor by 50% and28%, respectively. We also find that factors related to text extracted
from the commit logs prior to a release are the best predictors ofcrashing releases and have the largest effect.
Conclusions: Our proposed approach could help to identify crashreleases for mobile apps.

CCS Concepts•Software and its engineering → Software testing and debugging; •General and reference → Empirical studies; Evaluation;

Keywords
Mobile Applications, Crash Release, Prediction Model1.

INTRODUCTION

Mobile apps are some of the most popular software systems today. Recent market studies show that the Apple App Store has more∗

Corresponding Author

ACM ISBN 978-1-4503-2138-9.
DOI: 10.1145/1235

than 1.5 million apps and Google Play has over 1.6 million apps [2]. This rapid development of mobile app markets has broughthuge benefits. At the same time, competition in the mobile app
industry is very high since many apps provide similar functionality, e.g., there are more than 250 weather forecast applications in
Google Play1 .
Although mobile apps are considered to be software systems,
they are very different from traditional “shrink wrapped” software
such as Mozilla Firefox or Microsoft Windows [47]. For example,
most mobile apps tend to be small in nature and are developed bysmaller and less experienced teams [50]. More importantly, the roleof mobile apps has evolved from simple games that entertain us toplaying a critical role in our daily lives [40]. For example, mobile
apps are increasingly being used in mission-critical systems such as
security, healthcare and the financial sector. Therefore, the qualityof mobile apps is of critical importance.
At the same time, due to the fact that these mobile apps are released through app stores, mobile apps tend to have more releases
compared to traditional applications. To further enhance the competitiveness and attract more users, developers continually evolve
their applications and frequently release new versions. For example, the gothfox_ Tiny-Tiny-RSS app released 280 versions
in the 3 year period from September 2011 to July 2014. Other apps
have even shorter release cycles. However, in many cases, mobile developers may release versions of the app that are of poor
quality. Prior work showed that some of the most negatively impacting releases, are crashing releases, i.e., releases that cause theapp to crash [23]. These crashing releases can cause users to uninstall an app and potentially give it a negative rating [16], which inturn impacts the app’s revenues. Therefore, identifying crashingreleases early on, can help warn mobile app developers about a potential crashing version before it is released and reduce the numberof crashing releases. Notice our prediction is used to prioritize thereleases rather than replace testing.
In this paper, we perform an empirical study on crashing releasesof mobile apps. We perform our study on 10 open source mobile
apps from the from F-Droid repository2 . The 10 apps have 2,638
releases, of which 13% were crashing releases. For each app, we
mine its commit data to identify crashing releases. Since our goal isto predict crashing releases, we extract 20 different factors from themobile app repositories that are grouped into six unique dimensions: complexity, time, code, diffusion, commit, and text. Using theextracted factors, we leverage machine learning techniques to build
models that can effectively predict crashing releases. We compare
our approach with two baselines: random prediction and majority
prediction. Our results show that we are able to achieve an im1
2

https://play.google.com/store/search?q=weather%20forecast
http://f-droid.org/

Makes stats and browser deck-aware, and a fix crash caused bydeck selector

Figure 1: Example commit log between Releases 2.2alpha86
and 2.2alpha87 of the ankidroid_Anki-Android mobile app.
Prevent crashes in onPostExecute() by not calling
onCreateDialog(). Force startLoadingCollection() to re-sync with
AnkiDroidApp.getCol()

Figure 2: Example commit log between Releases 2.3alpha16
and 2.3alpha17 of the ankidroid_Anki-Android mobile app.
provement over random prediction by 50% and 28% in terms of F1and AUC score, and an improvement over majority prediction by
39% in terms of AUC score, respectively. Note that the precisionand recall for the majority prediction baseline are both 0, and thus
the F1-score for majority prediction cannot be calculated. In addition, we analyzed the factors that best indicate crashing releases
and find that factors in the text dimension, i.e., text in the commit
messages, are the best predictors.
The main contributions of the paper can be summarized as follows:

2.3alpha17 of the ankidroid_Anki-Android mobile app. We notethat in both cases presented in the Figures, the developers are mentioning fixes for crashes that were introduced by earlier versions.
Obviously these crashes were critical enough for the developers to
not only fix them, but to fix them quickly. In the example shownin Figure 1, the fix to the crashing release was performed in 5 days
and in the example shown in Figure 2, the fix was committed in 4
days.
Developer’s Opinions of Crashing Releases. As mentioned earlier, we also asked mobile developers at Hengtian3 and the developers of the mobile app projects ankidroid _Anki-Android
and qii_weiciyuan. We sent emails to 25 developers asking
them a very simple question:
Do you think it is necessary to predict the crash releases in mobile
apps? If so, why?
In total, 6 developers replied to our email. All the 6 developers,
denoted as D1 to D6, agreed that predicting crashing releases is an
important problem. We summarize their replies as follows:
1. Release Cycles: Since the release cycles for mobile apps arevery short (e.g., one release per week (D5)), “we do not haveenough time to test all aspects of the apps in each release”(D2). “If there is a tool that can tell us whether the releasehas a high likelihood of containing a crash, we will pay more
attention to the release and spend more time inspecting thecode” (D4).

1. We propose the problem of crashing release prediction formobile apps. To the best of our knowledge, this is the firstwork to predict crashing releases for mobile apps. We envision that our approach can be used to predict the releases thatare highly likely to be crashing so that they can be prioritizedfor testing.
2. We develop factors that can be easily mined from mobile app repositories to predict crashing mobile app releases. We
use the developed factors to accurately predict crashing releases and investigate the factors that best indicate crashingreleases.
3. We perform an empirical study on 10 open source mobile
apps and our experimental results show that our approachachieves an improvement over two other baseline approaches.
The remainder of the paper is organized as follows. We describe
the motivation of our study in Section 2. We present our empirical
study data and empirical study setup in Section 3 and 4, respectively. We present our empirical study results in Section 5. We discuss
additional points on the benefits and limitations of our approach in
Section 6. We discuss related work in Section 7. We conclude andmention future work in Section 8.

2.

MOTIVATION

Although our intuition and experience tells us that mobile app
developers have crashing releases and care about minimizing suchcrashing releases, we wanted to make sure that this was a real problem that mobile app developers face and care about (note that priorwork only showed that crashing releases negatively impact users [23]). Hence, we approached the task of investigating whether
crashing releases is a real problem in two ways. First, we examined the commit logs and found examples where developers explicitly mention that crashing releases occur and are swiftly fixed.
Second, we emailed a number of developers to get their opinion on
the importance of this problem. We elaborate on each of these twotasks in the following subsections.
Examples of Crashing Releases in Commit Logs. Figure 1
presents example commit logs between Releases 2.2alpha86 and2.2alpha87 of the ankidroid_Anki-Android mobile app. Figure 2
presents example commit logs between Releases 2.3alpha16 and2. Inspection Cost: In traditional desktop applications, foreach release, it may modify a large number of LOCs (e.g.,more than 1M LOCs). In such cases, predicting crashingreleases for desktop applications is not necessary since developers will need to inspect a large number of LOCs (D3).
However, “in mobile apps, in each release our team only
modified several thousand LOCs. If a tool can tell us there
is a crash in the release, it is possible for us to manuallyinspect the code” (D3). Note “it is impossible for us to inspect the code in all of the releases since we have too many
releases, and practically speaking, we can only focus on the
problematic releases” (D6).
3. Supplementary Means to Testing: Eventhough some apps
may have received enough testing, some of the crashes arestill hard to detect (D1). A crash release prediction tool canbe a supplementary means to testing (D1). After the testers
complete the testing, the tool can be used to ensure the quality of the released version.

3.

EMPIRICAL STUDY DATAThis section details the data used in our study. First, select therelevant mobile applications (application collection). Then, we identify the releases of the collected applications (release identification) and use heuristics to determine the crashing releases (crashrelease determination). We detail each step of the data collection
below.
Mobile App Collection: To perform our study, we need to obtaina number of mobile applications. To do so, we collected the mobile applications available on the Free and Open Source Software(FOSS) repository F-Droid. F-Droid is widely used in a number ofsoftware engineering studies (c.f., [1, 26, 29, 36]). Each mobile application available on F-Droid has a corresponding wiki page which
3

Hengtian is a large software outsourcing company in China,
with more than 2,000 employees. More details can be found in
http://www.hengtiansoft.com/?lang=en

Table 1: Statistics for the collected mobile applications. The
last row shows the total number of releases, the total numberof crashing releases, and the percentage of crash releases.
Figure 3: The manifest file of andstatus.
provides details about the app. For us, wiki pages are very important since they provided us with the number and dates of the releases for each app. In total, we crawled the data for 900 mobile apps.
This data included: 1) the wiki page of the app, 2) the link to thesource code repository, and 3) the source code of the app.
Since some of the mobile apps did not provide all of the information we required (e.g., there was no link to the source code repository available), we ended up with a total of 632 apps. Of these632 apps, 466 mobile apps are managed by git, 40 are managed bymercurial, and 126 are managed by svn. Since some of the tools
we used (e.g., the tools used to extract factors from commits) only
support Git repositories [49], we only consider the apps managed
by Git, i.e, 466 apps.
Release Identification: According to the Google Play developer
documentation4 , when a new version is released, the version attribute in the manifest file is also changed. More specifically, theandroid:versionCode and the android:versionName attributes must
be changed. Figure 3 presents an example manifest file from theandstatus app. The version code is “113”, and the version nameis “13.0”. With this information, we mine the commit logs in thesource code repository, and collect the dates of commit logs. We
use the dates from the commits and the dates for the version nameor version code in the manifest file (which we obtain from the wikipage), to determine which commits belong to a release.
To filter out toy examples and non-active apps, we manuallycheck the status of apps by reading their release notes and checking their download records. We select apps that existed in GitHub
for at least 2 years, since our prediction models require historical
data of the mobile apps. After this step, we are left with a total of
22 apps. Next, we further filtered out the apps that have less than
100 releases, since we require enough instances of crashing releases and non-crashing releases to train our models. This step further
narrowed the number of apps to 10. Although our requirement of
having more than 100 releases may seem restrictive, it is important
to note that our approach is most useful for apps with a large number of releases (apps with a small number of releases can afford tocheck every release).
Crashing Release Determination: Once we selected the 10 mobile apps, we extracted all releases for these apps. Then, for eachrelease, we extract all its associated commits along with the commit
messages from the Git repo of the project. To identify the crashing’releases, similar to prior work, we use keywords found in the commit messages [25,45,52]. In particular, we looked for the keywords“crash”, “crashed”, “crashing”, and “crashes”. Note that keyword
searches may increase the number of false positives and false negatives. To reduce the number of false positives, for each identified
crashing release, the first and third author manually checked thecommit logs and modified source code, and we only find 5 releases
are misclassified. To reduce the number of false negatives, for these
non-crash releases, the first and third author also randomly choose
and manually checked 30% ( i.e., 688)of the releases from these
non-crash releases, and we find all of these releases labeled as noncrash releases are correctly labeled. It is important to note herethat if we find these keywords in release x for example, then wemark release x − 1 as the crashing release. This is because developers will often mention that they are fixing a crash in a previous
release. Based on our observations, in practice crashing releases
4

https://developer.android.com/tools/publishing/preparing.html

Project

Time Period

#R

#C

%C

ankidroid_Anki-Android
bpellin_keepassdroid
BrandroidTools_OpenExplorer
freezy_android-xbmcremote
gothfox_Tinyguardianproject_Gibberbot
mariotaku_twidere
mtotschnig_MyExpenses
qii_weiciyuan
WSDOT_wsdot-android-app
Total

2009/06–2014/09
2009/01–2013/09
2011/12–2013/05
2009/08–2013/12
2011/09–2014/07
2010/07–2014/09
2012/04–2014/05
2011/05–2014/09
2012/07–2014/07
2010/05–2014/07

597
149
156
392
280
233
262
241
205
123
2,638

97
23
28
19
36
26
34
34
39
8
344

16%
15%
18%
5%
13%
11%
13%
14%
19%
7%
13%

are often followed by another release very swiftly. This is done tominimize the negative impact of the crashing release. Using themethod above, we identify all the potential crashing releases. Next,
we manually check the commit logs, fixed bugs, release notes anduser reviews of the crashing releases and their subsequent releases
to makes sure the releases identified as crashing in fact are.
Table 1 presents the statistics for each mobile app in our data set.
The columns correspond to the time period, the number of releases (# Releases), the number of crash releases (# Crash Releases),
and the percentage of the crash releases (% Crash Releases) for the
10 mobile applications. In total, we collect 2,638 releases acrossthe 10 mobile applications, and among them, 344 releases were labelled as crashing releases, which accounts for 13% of the releases,on average. Note that the number of crash releases is much less than
the number of the non-crash releases. We refer to this phenomenon
as the class imbalance phenomenon [18]. Due to the class imbalance phenomenon, we anticipate that predicting crashing releases
with high accuracy will be a difficult task.

4.

EMPIRICAL STUDY SETUP

Now that we have observed that the percentage of crashing releases is not negligible, the goal of our study is to be able to effectively predict crashing mobile releases early on so they can be
avoided. We extract a number of factors about the mobile releases
and use them to perform our prediction. We are interested in knowing how effective the extracted factors are at predicting crashingreleases. Furthermore, we determine the best indicators of crashing releases and examine their effect. We formalize our study in thefollowing research questions:
RQ1: Can we effectively predict crashing mobile releases?
RQ2: What factors are the best indicators of a crashing mobile release? What is the relationship of these factors with acrashing mobile release?

4.1

Factors Studied

To predict whether a release is a crashing release, we consider
20 factors grouped into six unique dimensions: complexity, time,code, diffusion, commit, and text. These factors are derived fromthe source control repository data of a mobile application. Table 2
presents the summary of the factors used in our study. We use
these 20 carefully developed factors because either 1) prior work
showed that they perform well in predicting defects [17, 19, 21, 35,
58, 60] or 2) intuitively, the factors made sense to include when
predicting a crashing mobile release. That said, we are the first toinvestigate the effectiveness of these factors in predicting crashingmobile app releases and encourage future factors to be developed
to help predict crashing mobile app releases. Next, we discuss eachdimension in detail.
Complexity Dimension: If the source code in a release has highcomplexity (e.g., high number of data flows in an applications), thecode will be harder to maintain, which may increase the chance ofTable 2: Factors used to identify crashing releases
Dimension

Name

Definition

Rational

Complexity

Cyclomatic

Time

PreDays

The number of branching paths within code in all the sourcecode files in a release.
The number of days since the previous release.

LA

Number of lines added in a release

LD

Number of lines deleted in a release

SIZE

Total number of lines of code in the current release

SAME
CUR_file

Number of source code files that are modified by both the current and the previous release
Number of source code files in the current releaseHigh complexity indicates that the code may be difficult to modify and has a higher chanceof containing/introducing a defect [42, 54].
The shorter the number of days passed from the previous releases, the high chance thecurrent release is a non-crashing release since developers need to fix the crashes appearedin the previous release.
The higher the number of lines added, the more likely a defect can be introduced, increasing
the chance of a crash release [39, 41].
The higher the number of lines deleted, the higher chance that a previous bug recurs, orimportant functionality is removed [39, 41].
An increase in the SIZE causes an increase in complexity, and high complexity may resultin crash releases [39, 41].
The higher the number of the same source code files that are modified, the higher the chancethat the current release fixes existing defects in the previous release.
An increase in the number of source code files between two releases (CUR_file PREV_file) may indicate that the current release adds more functionality and/or performs
many fixes, which may result in crash release.

PREV_file
Top_NS

Number of modified source code files in the previous release
Number of unique subsystems changed between two releases

Bottom_NS
NF
File_entropy

Number of unique subsystems changed between two releases
Number of unique files that have changed between two releases
Distribution of modified files across the release

Churn_entropy
NC

Distribution of modified code across the application
Number of commits

NFC

Number of commits which fix bugs

Fuzzy_score

Fuzzy set scores of commit logs

NB_score
NBM_score
DMN_score
COMP_score

Naive Bayes scores of commit logs
Naive Bayes Multinomial scores of commit logs
Discriminative naive Bayes Multinomial scores of commit logs
Complement naive Bayes scores of commit logs

Code

DiffusionCommit5

https://scitools.com/

Releases that touch too many files have a higher chance of being a crash releases.
Releases with high entropy are more likely to be crash releases, since a developer needs toinspect large number of scattered changes (at file or code levels) across each file.
The more number of commits, the more functionality added and/or defects fixed. This may
increase the chance of a crash release.
The higher the number of commits that fix defects, the higher the chance that a future defectmay be introduced, leading to a higher chance of a crash release [25, 45].
The text content of commit logs between the previous release and the current release. We
represent the text of the commit logs using fuzzy set scores [65], naive Bayes scores [34],naive Bayes multinomial scores [34], discriminative naive Bayes multinomial scores [53],and complement naive Bayes scores [46]. A higher score means the release is more likely
to be a crash release.

Text

a crashing release. Also, the defect prediction literature showedthat complexity (e.g., cyclomatic complexity) is a good predictorof defect-prone modules [42, 54]. We expect that the complexity
dimension can be used to determine the likelihood of a crashingrelease. As shown in Table 2, we propose McCabe’s cyclomatic complexity in the complexity dimension. McCabe’s cyclomatic
complexity can be measured directly from the source code in thecurrent release using standard code analysis tools. In our study weused the Understand tool5 to determine the code complexity.
Time Dimension: If the time period between the previous releaseand the current release is short, the current release has a higher
chance to be a non-crash release since it may fix the crashes appeared in the previous release. We expect that the time dimension
can be used to determine the likelihood of a crashing release. As
shown in Table 2, we propose the number of days since the previous release (PreDays) to make up time dimension, PreDays iscomputed by counting the number of days between the previous
and current release.
Code Dimension: If a release has large changes done to its source
code, then it has a higher probability to introduce more defects,
which in turn may cause a crashing release [42]. Previous defectprediction studies show that the size of changes (e.g., number oflines of code added or deleted) is a good indicator of defects [39,41]. Also, if the current release modifies many of the same sourcecode files as the previous release, this may be an indication that a lotof fixing is being done, which may indicate that the current releaseis a good release. We expect that the code dimension can be used todetermine the likelihood of a crashing release. As shown in Table 2,
we propose 6 factors which make up the code dimension. These
factors can be measured directly from the source control repository
by comparing the difference between 2 releases.
Diffusion Dimension: In general, a highly distributed release ismore difficult to understand, and requires more work to inspect all

The more functionalities there are in an application release, the more prone it is to fail. We
use the subsystem as a proxy for a feature. Releases that contain many modifications at thesubsystem level are more likely to be crash releases.

the locations that are changed. Also, previous defect prediction studies found that the number of subsystems touched is an indicatorof defects [38], and scattered changes are good indicators of defects [17]. We expect that the diffusion dimension can be used todetermine the likelihood of a crash release. As shown in Table 2,
we propose 5 factors that make up the diffusion dimension.
We use the top directory name and bottom directory name as thesubsystem name to measure Top_NS and Bottom_NS, respectively.
For example, if a commit changes a file in the path“src/app/easytoken/MainActivity.java”, then its top directory nameis “src/”, and the bottom directory name is “src/app/easytoken/”.
For the ith release, we compute the set of top directory name T opi
and bottom directory name Bottomi . Then, for two consecutive
releases (ith and (i + 1)th releases), Top_NS= |T opi ∩ T opi+1 |,and Bottom_NS =|Bottomi ∩ Bottomi+1 |.
To measure the file and churn entropy, we use the measures similar to the measures
P proposed by Hassan [17]. Entropy is computed
as: H(P ) = − n
k=1 (pk × log2 pk ). In the above equation, n is
the number of files changed in
Pthe release, pk ≥ 0 is the probability
for file k, and pk satisfies ( nk=1 pk ) = 1. pk can be computed
at different levels of granularity: to compute file entropy, pk is the
proportion that file k is modified in the changes6 in a release, tocompute churn entropy, pk is the proportion that the number oflines of code in file k that is modified in the changes in a release.
Entropy aims to measure the distribution of the release across the different files or the lines of code in the files. The higher the entropy,
the larger the spread of a release.
Commit Dimension: If a release has a large number of commits,
we conjuncture that the release has a high probability to be a crashing release. This belief is driven by the intuition that more commits
mean more changes (e.g., bug fixes, new functionalities) to the application, which may introduce more problems (e.g., bugs) in the
6
In a release, there are multiple commits, and each commit corresponds to a change.

release. As shown in Table 2, we propose 2 factors which makeup the commits dimension: the number of commits (NC), and thenumber of bug fixing commits (NFC). NC is computed by counting
the number of commits in the current release, and NFC is computed by counting the number of commits which contains the strings“fix”, “’error”, “fault”, “crash”, “issue”, or “bug” [25, 45, 52].
Text Dimension: In the text dimension, we extract a number of
textual factors from commits, and we convert the terms that appear
in commit logs into numerical values. In total, we compose 5 different factors in the text dimension, which correspond to the textual
scores based on 5 different classifiers.
During the release of an application, many commits may be submitted to fix defects or implement new features. Since prior work
showed that text descriptions can help with bug prediction [60], we
believe that analyzing the natural language description in the commit logs may help us identify crashing releases. We refer to the factors extracted from the commit logs as textual factors. To extract
the textual factors, we first tokenize the text in the commit logs
into words, phrases, symbols, or other meaningful name element
tokens. Next, we remove the stop-words such as “I”, “the”, “he”,
which carry little value to discriminate crashing releases. Then, we
perform stemming on the tokens, which reduces inflected (or sometimes derived) tokens to their stem, base or root form (e.g., “write”,“wrote”, and “written” are reduced to the root form “writ”). We
use the resulting textual tokens and count the number of times eachtoken appears in the commit logs of a release.
Our textual dataset contained a large number of tokens. Since a high number of tokens (factors) can cause, what is known
as the curse-of-dimensionality [15], we follow previous studies(e.g., [60]) and we convert the textual factors into numerical values,
which associate each textual feature with a value that it indicates acrashing releases and another value that the textual feature is indicative of a non-crashing release. For example, the word bug may
have a score x that it indicates a crashing release and a score y thatit indicates a non-crashing release.
To come up with the scores for the different textual features, wefirst divide our data into a training set and a testing set. Then, thetraining data set is split into two training subsets by leveraging stratified random sampling, so that the distribution and number of
non-crashing and crashing releases in both training subsets is thesame [60]. Next, we extract the textual factors from the training
subsets, and create word frequency tables based on the extracted
factors. We train a classifier with the first training subset, and use it
to obtain the textual scores on the second training subset. We also
train a classifier with the second training subset, and use it to obtainthe textual scores on the first training subset. We use this strategy to
avoid the textual classifiers from being biased toward their training
sets [60]; otherwise, our models may lead to optimistic (unrealistic) values for the textual scores. In the prediction phase, for a newrelease, we leverage the text mining classifiers that are built on allof the training releases to compute the values of the textual factors.
In this paper, we use 5 types of textual classifiers to calculate
the scores of the textual features, which are: fuzzy set classifier [65], naive Bayes classifier [34], naive Bayes multinomial classifier [34], discriminative naive Bayes multinomial classifier [53],and complement naive Bayes classifier [46], and we denote thetextual scores using these 5 classifiers as fuzzy_score, NB_score,
NBM_score, DMN_score, and Comp_score respectively. In thefollowing paragraphs, we present the details of the 5 classifiers.
Fuzzy Set Classifier: A fuzzy set classifier considers the concurrence of words and labels (i.e., crashing or non-crashing release) as
good indicators of identifing crashing releases. If a word always ap-

pears in a crashing release in the training set, then for a new release,
if the same word appears, the release will be assigned a high probability of being a crashing release. We denote the label of the ith
release as li , and we represent the textual factors in the ith releaseas a vector of weights denoted by Ri = hft1 ,i , ft2 ,i , · · ·, ftm ,i i
(ftn ,i denotes the nth factor of Ri ), where ftn ,i = 1 represents
that the word token tn appears in the commit logs of the ith release; and ftn ,i = 0 else wise. Based on these notations, we definea label-term affinity score as follows:
D EFINITION 1. (Fuzzy Label-Word Affinity Score.) Consider a historical release data collection D, and a set of labels L.
For each label l ∈ L, and word token t ∈ D, the fuzzy labelword affinity score of l and t, denoted as Aff (l , t), is computednas Aff (l , t) = nl +nwl,t−nl,t . nl,t denotes the number of releases
whose commit logs contain the word token t and are of the labell, nl denotes the number of releases that have the label l and nt
denotes the number of releases that contain the word token t.
In our paper, we have two labels, crashing release C and noncrashing release N C. Thus, for each word token t ∈ D, we have
two fuzzy label-word affinity scores, Aff (C , t), and Aff (NC , t).
For a new release RN ew = hft1 ,new , ft2 ,new , · · ·, ftm ,new i, wedefine the label-release affinity score as follows:
D EFINITION 2. (Label-Release Affinity Score.) For a newrelease RN ew , the label-release affinity score F uzzy(l, N ew) is
computed as the combinations of the label-word affinity scoresAf
Q f (l, t) of its associated terms w: F uzzy(l, N ew) = 1 −t∈N ew (1 − Af f (l, t)). t denotes the word tokens that appear
in the commit logs of the new release.
For a new release, we have 2 label-release affinity scores,
i.e., F uzzy(C, N ew) and F uzzy(N C, N ew), the fuzzy score(Fuzzy _score) to be a crash release for fuzzy set classifier is
F uzzy(C,N ew)
F uzzy(N ew) = F uzzy(C,N.
ew)+F uzzy(N C,N ew)
Naive Bayes Classifier: In Naive Bayes classifier [34], we consider
the number of times a word as good indicators to identify crashingreleases. Previous studies show that Naive Bayes is a fast and effective text classification technique [34]. Similar to the fuzzy set
classifier, we represent the textual factors in the ith release as abinary vector denoted by Ri = hft1 ,i , ft2 ,i , · · ·, ftm ,i i (ftn ,i denotes the nth factor of Ri ), where ftn ,i = 1 represents that theword token tn appears in the commit logs of the ith release; andftn ,i = 0 else wise. For each word token tn , we count the numberof times that tn appears in the crashing releases as Crashn , andthe number of times that tn appears in the non-crashing releases as
N onCrashn . Then, the probability that wj is related to crashingnreleases as P (tn ) = CrashnCrash.
+N onCrashn
After the naive Bayes classifier is built, we compute the NB_scoreQof the commit logs in a new release N ew as N B(N ew) =P
Q
Q(tn ), where tn refers to the word in the commit
P (tn )+ (1−P (tn ))logs of N ew.
Naive Bayes Multinomial Classifier: Naive Bayes multinomial(NBM) is one of the variants of naive Bayes algorithm which builds
a classifier based on multinomially distributed data [34]. McCallum et al. empirically find that NBM performs better than Naive
Bayes in text classification tasks [34]. Different from Naive Bayes,
NBM leverages the word frequency information to perform textclassification, i.e., we represent the textual factors in the ith releaseas a vector of weights denoted by Ri = hwt1 ,i , wt2 ,i , · · ·, wtm ,i i,where wtn ,i is the number of times the word token tn appears in
the commit logs of the ith release. We use the implementation ofnaive Bayes multinomial on top of Weka [14]. For a new release
RN ew , we denote the likelihood score to be a crash release of thenaive Bayes multinomial classifier as NBM_score N BM (N ew).

Discriminative Naive Bayes Multinomial Classifier: Discriminative naive Bayes multinomial (DMN) is one of the variants of naive
Bayes algorithm multinomial which learns parameters by discriminatively computing frequencies from data [53]. Su et al. empirically find that compared with Naive Bayes, DMN converges quickly,
and does not suffer from the over-fitting problem [53]. We use theimplementation of DMN on top of Weka [14]. For a new release
RN ew , we denote the likelihood score to be a crash release of thediscriminative naive Bayes multinomial classifier as DMN_score
DM N (N ew).
Complement Naive Bayes Classifier: Complement Naive Bayes
Classifier (COMP) is one of the variants of Naive Bayes multinomial (NBM) algorithm which uses various transformation approaches in information retrieval community to improve the prediction performance for textual data [46]. Different from Naive Bayes
multinomial that assumes the text follows multinomial distribution.
COMP proposes the usage of heuristic solutions (i.e., term frequency, document frequency, and document length transformation) tomodel better text [46]. We use the implementation of complement
naive Bayes on top of Weka [14]. For a new release RN ew , wedenote the likelihood score to be a crash release of the complement
naive Bayes classifier as Comp_score COM P (N ew).

4.2

Prediction Model

For each of our empirical study projects, we use our proposed
factors to train a Naive Bayes classifier to predict whether a release
will be a crashing release or not. We also compare our prediction
model with four other classifiers namely: decision tree, kNN, andRandom Forest.
Naive Bayes (NB): We use the Naive Bayes classifier for two purposes: to convert textual information into numerical values (i.e., toobtain the probability that the textual contents in the commit logs
are related to crash release), and to build a prediction model thatpredicts if a release will be a crashing release or not.
Decision Tree (DT): C4.5 is one of the most popular decision tree
algorithms [15]. A decision tree algorithm classifies data points
by comparing their factor with various conditions captured in thenodes and branches of the tree.
K-Nearest Neighbor (kNN): K-nearest neighbor is an instancebased algorithm for supervised learning, which delays the induction or generalization process until classification is performed [15].
We use the Euclidean distance as the distance metric, and since theperformance of kNN may be impacted by different values of k, we
set k from 1 to 10, and report the best performance (in terms ofF1-score) among the 10 values of k.
Random Forest (RF): Random forest is a kind of combination approach, which is specifically designed for the decision tree classifier [5]. The general idea behind random forest is to combine
multiple decision trees for prediction. Each decision tree is built
based on the value of an independent set of random vectors. Random forest adopts the mode of the class labels output by individual
trees.

4.3

Dealing with Data Imbalance

As can be noted, the number of crashing releases is much lessthan non-crashing releases (i.e., we have a major class imbalancein the data). For example, in WSDOT_ wsdot-android-app,
only 9% of the releases are crashing releases. In this paper, we
perform both over- and under-sampling to alleviate the imbalanceissue. In over-sampling, we increase the number of the minority class instances (in our case, crashing release). And in undersampling, we decrease the number of the majority class instances(in our case, non-crashing releases). Previous studies (e.g., [8])recommend that we perform both under- and over-sampling, since
under-sampling may lead to useful data being discarded and oversampling may lead to over-fitted models. We perform both overand under-sampling on the training data and predict using a nonbalanced test data set (i.e., we do not resample the testing data).
We use the resampling algorithm in Weka [14] to do the over- andunder-sampling on the training data, and we use the option “-B 1.0”
(ensure the class distribution is uniform in the output data), and “Z 100” (the final sample size is the same as the original dataset).
After the resampling, the number of crash releases and non-crashreleases are the same in the training set.

4.4

Experimental Evaluation

There are four possible outcomes for a release in the test data: arelease is classified as a crashing release when it truly is a crashingrelease (true positive, TP); it can be classified as a crashing release
when it is actually a non-crashing release (false positive, FP); it can
be classified as a non-crashing release when it is actually a crashingrelease (false negative, FN); or it can be classified as a non-crashingrelease and it truly is a non-crashing release (true negative, TN).
Based on these possible outcomes, precision, recall and F1-scoreare defined as:
Precision: is the proportion of releases that are correctly labeled
as crashing releases among those labeled as crashing releases, i.e.,
P = T P/(T P + F P ).
Recall: is the proportion of crashing releases that are correctly labeled, i.e., R = T P/(T P + F N ).
F1-score: is a summary measure that combines both precision andrecall - it evaluates if an increase in precision (recall) outweighs a
reduction in recall (precision), i.e., F = (2×P ×R)/(P +R). F1score is the harmonic mean of precision and recall, which is usedin many software analytics studies [19, 24, 43, 56, 62].
AUC: In addition to the F1-score, we also use the Area Under the Receiver Operating Characteristic Curve (AUC) to evaluate the effectiveness of our approach. AUC is a commonly-usedmeasure to evaluate classification performance, and many othersoftware engineering studies also use AUC as an evaluation metric [27, 28, 48, 55]. The larger the AUC is, the better is the performance of a classification algorithm.

5.

EMPIRICAL STUDY

In this section, we answer the research questions posed earlier7 .
RQ1: Can we effectively predict crashing mobile releases?
Motivation: In order to avoid a negative user experience and poor
ratings, we would like to effectively identify crashing releases early on so they can be avoided (or more quality assurance efforts canbe targeted towards such releases). Therefore, we use our proposed
factors and build prediction models to examine whether it is feasible to build accurate models that help to predict crashing releases.
Approach: We implement our proposed approach on top of theWeka tool [14]. We use 100 times stratified 10-fold cross validation to estimate the accuracy of our models. In stratified 10-foldcross validation we randomly divide the dataset into ten folds. Of
these ten folds, nine folds are use to train the classifier, while theremaining one fold is used to evaluate the performance. The class
distribution in the training and testing datasets is kept the same as
the original dataset to simulate real-life usage of the algorithm. We
run 10-fold cross validation 100 times to further reduce the bias due
to training set selection. To evaluate their performance, we use theprecision, recall, F1, and AUC metrics. The reported performance
7
The datasets used in this paper can be downloaded in https://www.dropbox.com/s/hr6amcyssuj194c/dataset.zip?dl=0

Table 3: Precision, recall, and F1-score for our approach (our) compared with the baseline approaches. The best F1-score and AUC
values are highlighted in bold.
Projects
ankidroid_Anki-Android
bpellin_keepassdroid
BrandroidTools_OpenExplorer
freezy_android-xbmcremote
gothfox_Tiny-Tiny-RSS-for-Honeycomb
guardianproject_Gibberbot
mariotaku_twidere
mtotschnig_MyExpenses
qii_weiciyuan
WSDOT_wsdot-android-app
Average

Naive Bayes Model (our)Precision
Recall
F1
AUC
0.18
0.18
0.22
0.21
0.15
0.12
0.20
0.17
0.41
0.19
0.20

0.52
0.61
0.79
0.58
0.56
0.73
0.50
0.68
0.67
0.63
0.62

0.26
0.28
0.34
0.31
0.24
0.20
0.29
0.27
0.51
0.29
0.30

0.50
0.61
0.70
0.70
0.60
0.52
0.63
0.59
0.78
0.78
0.64

of the models are the average of the 100 times stratified 10-foldcross validation. These metrics are compared to the performanceof our baseline model.
Here, we choose two baseline models: random prediction, andmajority prediction. In random prediction, it randomly predicts crash releases. The precision for random prediction is the percentage of crash release in the data set. Since the random prediction model is a random classifier with two possible outcomes (e.g.,crash/non-crash release), its recall is 0.50. Majority prediction always predicts the label of an instances as the majority class (in ourcase, majority prediction will predict every release as a non-crashrelease). In majority prediction, its T P will be 0 since none of thereleases are predicted as crash releases, and thus its precision andrecall are both 0, which in turns cause the F1-score to be NA. To
compute the improvement of our approach over the baseline approaches, we denote the evaluation metric score of our approach as
our, and the score of the baseline approach as baseline. Then, the× 100%.
improvement is computed as our−baseline
baseline
Results: Table 3 presents the precision, recall, F1, and AUC scoresfor our approach compared with the baseline approach for the 10
mobile apps, respectively. On average across the 10 projects, ourapproach achieves precision, recall, F1-score, and AUC scores of
0.20, 0.62, 0.30, and 0,64, respectively. Our approach improves
over the random prediction by 54%, 24%, 50%, and 28% in terms
of precision, recall, F1-score, and AUC scores, respectively. Also, compared with majority prediction, considering its precision,
recall, and F1 scores are 0, 0, and NA, our approach improve the
AUC score of majority prediction by 39% on average across the 10
projects.
We notice for freezy_android-xbmcremote and WSDOT_wsdotandroid-app, the improvements of our approach over random prediction are 251% and 156% in terms of F1-score. We find that
only 5% and 7% of releases are crash releases in these two datasets, i.e., these two datasets are more imbalanced than the otherdatasets. This indicates that our approach is best suited for highly imbalanced cases. To test whether the improvement of our approach over random prediction on highly imbalanced datasets (i.e.,
freezy_android-xbmcremote and WSDOT_wsdot-android-app) arehigher than these on the other datasets is statistically significant, weapply Wilcoxon Rank Sum test [61], and the p-value we achieve is
0.0222. This indicates that the improvement is statistically significant at the confidence level of 95%. Moreover, we also use Cliff’sdelta [6], which is a non-parametric effect size measure that quantifies the amount of difference between the two groups. The Cliffaŕs
˛delta is 0.875, which corresponds to a large effect size.
Considering that only a small proportion of releases are crash release (on average across the 10 apps, 13% of the releases are crashreleases), the data imbalance phenomenon makes the prediction ofcrashing releases a hard task. Some previous studies on software
engineering (c.f., [57, 59, 63]) also achieved similar F1-scores, andin the future, we plan to improve the F1-score of our approach further by designing a better algorithm and considering more factors.

Random Prediction Model
Precision
Recall
F1
AUC
0.16
0.15
0.18
0.05
0.13
0.11
0.13
0.14
0.19
0.07
0.13

0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50

0.25
0.24
0.26
0.09
0.20
0.18
0.21
0.22
0.28
0.12
0.20

Majority Prediction Model
Precision
Recall
F1
AUC

0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA

0.49
0.45
0.46
0.47
0.46
0.45
0.46
0.46
0.48
0.39
0.46

Table 4: Top-5 factors which achieve the highest IncNodePurity(Inc.) and effect scores
FactorFuzzy_sc.
Cyclomatic
DMN_sc.
LA
CUR_file
Factor
NBM_sc.
Cyclomatic
CUR_file
SAME
LA
FactorFuzzy_sc.
LA
NBM_sc.
Chrun_en.
LD
Factor
NBM_sc.
Fuzzy_sc.
SAME
SIZE
LD
Factor
DMN_sc.
Cyclomatic
NBM_sc.
Fuzzy_sc.
Chrun_en.

Inc.
7.46
7.22
6.88
4.79
4.30

andkidroid.
Effect
1.00
0.23
0.21
0.21
0.22

Inc.
2.28
1.78
1.65
1.29
1.28

Brandroid.
Effect
1.00
0.99
0.99
1.00
1.00

Inc.
3.80
2.47
2.32
1.75
1.62

gothfox.
Effect
0.93
0.43
0.43
0.44
0.44

Inc.
2.91
2.55
1.91
1.84
1.56

mariotaku.
Effect
0.02
0.02
0.02
0.02
0.02

qii_weiciyuan.
Inc.
Effect
2.62
0.74
5.79
0.06
1.66
0.01
1.59
0.01
1.14
0.01

Factor
DMN_sc.
Cyclomatic
PreDays
CUR_file
NBM_sc.
Factor
DMN_sc.
LA
Cyclomatic
Fuzzy_sc.
SAME
Factor
NBM_sc.
LA
Chrun_en.
CUR_file
Fuzzy_sc
Factor
CUR_file
DMN_score
Chrun_en.
Cyclomatic
LA
Factor
DMN_sc.
NBM_sc.
Fuzzy_sc
Cyclomatic
Comp_sc.

Inc.
1.46
2.29
1.42
1.30
1.29

bpellin.
Effect
0.80
0.44
0.22
0.23
0.23

Inc.
1.99
1.69
1.64
1.11
0.81

Freezy.
Effect
1.00
0.00
0.00
0.02
0.00

Inc.
2.06
1.96
1.54
1.50
1.25

guardian.
Effect
0.99
0.99
0.99
0.98
0.86

Inc.
2.33
2.03
1.79
1.76
1.67

mtotschnig.
Effect
0.85
0.70
0.88
0.87
0.82

Inc.
0.92
1.33
0.82
0.60
0.50

WSDOT.
Effect
0.04
0.00
0.00
0.00
0.00

RQ2: What factors are the best indicators of a crashing mobile release? What is the relationship of these factors with acrashing mobile release?
Motivation: In addition to identifying crashing releases with highaccuracy, we are interested in knowing what factors are good indicators of crashing releases and the relationship of the different
factors with crashing releases. Knowing which and by how mucheach factor relates to a crashing release helps practitioners determine what factors they should carefully consider when determining
which releases have a higher chance of being a crashing release.
Approach: To determine the important factors in determining acrashing release, we use IncNodePurity scores in the output of the
R random forest library [11]. IncNodePurity refers to the mean decrease in node impurity, i.e., a higher IncNodePurity score meansthat the corresponding factor plays an more important role in thebuilt model. Since our goal here is understanding and not prediction, we use the entire dataset to build the random forest models
since we only need to use IncNodePurity scores outputted by random forest.
In addition, similar to prior work [37, 51], we measure the relationship (i.e., effect) of each factor with a crashing release. To doso, we create a model where factors are set to their median values,
we predict the likelihood score for the release to be a crashing re-lease, and denote these likelihoods as base. Then, for each factor
f , we create a model where we double the median of the factor f ,while keeping all other factors at their median values. We computethe likelihood score for the release to be a crashing release, anddenote the score as fcrash . We measure the effect of the factor f−base)
by using ef f (f ) = (fcrash. The effect of a factor can bebase
positive or negative. A positive effect indicates that a higher levelof a factor corresponds to an increase in the likelihood of a release
being a crashing release, while a negative effect indicates that ahigher level of a factor corresponds a decrease in the likelihood ofa release being a crashing release.
Results: Table 4 presents top-5 factors which achieve the highest
IncNodePurity scores of factors as assigned by the random forest,
and the effect scores. We notice for different datasets, the most important factors are different. For example, the top-2 important factors are fuzzy_score and Cyclomatic in ankidroid_Anki-Android,
and the top-2 important factors are NBM_score and fuzzy_score inmariotaku_twidere.
As for the relationship of the different factors with the prediction of a crash release, we find that all of the 20 factors havethe non-negative effect with the crash releases. For BrandroidTools_OpenExplorer and guardianproject_Gibberbot, we noticethe effect scores are much higher than the others, which means that in these apps a higher level of a factor corresponds toan increase in the likelihood of a release being a crashing release. For freezy_android-xbmcremote, mariotaku_twidere, and
WSDOT_wsdot-android-app, the effect scores are much lower thanthe others, which means the impact of these factors to identify crashreleases is low.
Considering both the importance (i.e., IncNodePurity) and effect of the factors, we find the factors in the text dimension suchas Fuzzy_score, NB_score, NBM_score, DMN_score, and Comp_score show higher IncNodePurity and effect score compared toother factors. For example, DMN_score shows the highest importance and effect scores for freezy_android-xbmcremote and qii_weiciyuan.
To investigate why factors in the text dimension are more important and effective than the other factors, we also manually analyze the text in the commit logs. For example, in ankidroid_AnkiAndroid, the Fuzzy_score is one of the most important factors. For
each term, we get its Fuzzy_score, and we rank these scores fromhigh to low. We find the top-5 terms with the highest Fuzzy_scoresare “screen”, “loading”, “UI”, “button”, and “view”, and theseterms are all related to problems in UI.
Similarly, in BrandroidTools_OpenExplorer, the NBM_score is
one of the most important factors. We also rank the terms according
to their NBM_scores, and we find the top-5 terms with the highest NBM_scores are “dialog”, “translation”, “icon”, “page”, and“box”. The terms “dialog”, “icon”, “page”, and “box” are related
to UI issues, and the term “translations” is related to the languagetranslation. We further investigate the patches for the crash releases, we find in BrandroidTools_OpenExplorer, a number of crashreleases are caused by the enhancement or modification of UI, andalso a number of crash releases are caused due to language translation (such as translate the language to French, Japanese, or Korean).
For the other apps, we also check the factors in the text dimension, and we find that a number of crashes are caused by UI enhancement or modification. In practise, users cannot bear the crashes in UI, and our finding is in consistent with the practical user
experience.
Notice that in our paper, to ensure our prediction model is notbiased by the large number of textual features, we only use 5 numeric textual factors (we initially start with 20 factors) by using 5

Table 5: Prediction models by using different classifiers. NB=Naive Bayes, RF=Random Forest, C4.5=Decision Tree. The
best precision, recall, F1 and AUC scores are in bold.
Algo.
NB
kNN
RF
C4.5
Algo.
NB
kNN
RF
C4.5
Algo.
NB
kNN
RF
C4.5
Algo.
NB
kNN
RF
C4.5
Algo.
NB
kNN
RF
C4.5

Prec.
0.18
0.17
0.22
0.18

ankidroid
Recall
F1
0.52
0.26
0.51
0.26
0.28
0.24
0.36
0.24

AUC
0.50
0.50
0.57
0.51

Prec.
0.18
0.16
0.18
0.14

bpellin.
Recall
F1
0.61
0.28
0.53
0.25
0.30
0.23
0.35
0.20

AUC
0.61
0.50
0.49
0.45

Prec.
0.22
0.19
0.21
0.26

Brandroid.
Recall
F1
0.79
0.34
0.57
0.29
0.29
0.24
0.50
0.35

AUC
0.70
0.51
0.61
0.62

Prec.
0.21
0.07
0.11
0.05

freezy.
Recall
F1
0.58
0.31
0.53
0.12
0.11
0.11
0.16
0.08

AUC
0.70
0.62
0.57
0.51

Prec.
0.15
0.16
0.28
0.22

gothfox.
Recall
F1
0.56
0.24
0.69
0.26
0.36
0.32
0.39
0.28

AUC
0.60
0.56
0.65
0.59

Prec.
0.20
0.18
0.33
0.21

mariotaku.
Recall
F1
0.50
0.29
0.59
0.27
0.44
0.38
0.41
0.28

AUC
0.63
0.63
0.69
0.62

Prec.
0.17
0.17
0.18
0.23

mtotschnig.
Recall
F1
0.68
0.27
0.57
0.26
0.21
0.19
0.47
0.31

AUC
0.59
0.62
0.59
0.62

Prec.
0.41
0.29
0.38
0.38

qii_wei.
Recall
F1
0.67
0.51
0.69
0.41
0.49
0.43
0.59
0.46

AUC
0.78
0.70
0.77
0.67

Prec.
0.19
0.23
0.30
0.25

WSDOT.
Recall
F1
0.63
0.29
0.75
0.35
0.38
0.33
0.38
0.30

AUC
0.78
0.85
0.82
0.65

Prec.
0.20
0.17
0.23
0.21

Average.
Recall
F1
0.62
0.30
0.60
0.26
0.30
0.26
0.39
0.27

AUC
0.64
0.60
0.63
0.58

different text mining techniques. Moreover, besides the text factors, other factors such as Cyclomatic and LA are also importantfactors to predict crashing releases. For example, Cyclomatic and
LA appear in the top-5 most important factors for 6 out of 10 apps.

6.

DISCUSSION

Comparison Using Different Classifiers: Besides the Naive
Bayes classifiers, there are other popular machine learning algorithms that can be used to predict whether a release is a crashing release or not. In this section, we compare the performanceof three other classifiers namely: kNN, decision tree and random
forest. Table 5 presents the precision, recall, F1, and AUC scores
achieved by these classifiers. We notice different classifiers show
different performance in different apps. For example, random forest shows the best F1 and AUC scores in gothfox_Tiny-Tiny-RSSfor-Honeycomb and mariotaku_twidere, and kNN shows the best
F1 and AUC scores in WSDOT_wsdot-android-app. On average
across the 10 apps, we find that Naive Bayes achieves the best
performance. Thus, in practice, we recommend developers to use
Naive Bayes when predicting crashing releases.
Longitudinal Data Setup: To investigate whether our tool can beused to solve the problem in the same setting as the one in practice,
we performed an experiment using a longitudinal data setup (i.e.,
not using cross validation). We sorted the releases in the temporalorder they are published. Then we built a prediction model by using
the first 70% of the releases, and predict the labels for the remaining
30% of releases. Note that in WSDOT_wsdot-android-app, there
are no crashing releases in the latest 30% of releases, thus the F1
and AUC scores are NA. Table 6 presents the precision, recall, F1,
and AUC scores for Naive Bayes in longitudinal data setup. We
see that Naive Bayes, in longitudinal data setup, achieves similar
F1 and AUC scores as Naive Bayes in cross-validation setting. The
average F1 and AUC scores for Naive Bayes in longitudinal data setup are 0.28 and 0.62, while these scores for Naive Bayes incross-validation are 0.30 and 0.64.

Table 6: Precision, Recall, F1, and AUC scores for Naive Bayesin longitudinal data setup.
Project

Precision

RecallF1

AUC

ankidroid_Anki-Android
bpellin_keepassdroid
BrandroidTools_OpenExplorer
freezy_android-xbmcremote
gothfox_Tiny-Tiny-RSS-for-Honeycomb
guardianproject_Gibberbot
mariotaku_twidere
mtotschnig_MyExpenses
qii_weiciyuan
Average.

0.11
0.31
0.23
0.06
0.15
0.18
0.28
0.22
0.33
0.21

0.22
0.79
1.00
0.88
0.50
0.58
0.65
0.53
0.15
0.59

0.15
0.45
0.37
0.12
0.23
0.28
0.39
0.31
0.21
0.28

0.50
0.56
0.81
0.74
0.57
0.59
0.62
0.55
0.60
0.62

Precision vs. Recall: The goal of our approach is to flag crashing releases to improve the quality of mobile apps. As with anyclassification technique, there is a tradeoff between precision andrecall. As we have shown, our approach easily outperforms thebaseline approaches in terms of F1-measure and AUC, however, it
is important to note that we tend to achieve better recall than precision. The reason for favouring recall over precision is due to thefact that crashing releases are very negatively perceived by users,
hence, it is important that we avoid crashing releases at any cost.
That said, it is important to note that one can easily modify thethreshold used in our technique (currently we set the threshold tobe 0.5), to achieve a higher precision at the cost of lower recall. Depending on individual/project-specific circumstances, users of ourtool can adjust the threshold based on how much recall or precision
matter to them.
Usefulness of Our Proposed Tool: After we completed the study,
we also sent a version of this paper to mobile app developers in acompany, called Hengtian, to inquire about the usefulness of ourapproach and the factors used to perform our prediction. In total,
we sent the paper to 20 developers, and we received 12 responses(denoted as R1 to R12). The majority of the developers (11 out of
the 12) consider the proposed factors and machine learning techniques to be practical and applicable in a real mobile development
environment. For example, R2 states: “we can easily extract thesefactors from the commits in a release, and also the naive Bayesclassifier is understandable”. However, R5 has some questions about how machine learning techniques could work in practice.
Moreover, 10 out of the 12 developers agree that the results showthat our proposed approach is useful. For example, R9 states that“although F1-score is not high, the recall score is good, in practice we are interested to find all the crash releases as possible, thus
recall is more important than precision. The proposed tool could
enhance the confidence about the quality of a release”. On the other hand, two developers (R5 and R7) wish the F1-score would behigher. Therefore, in the future we plan to continue investigatingother factors that can help improve our prediction accuracy.
Threats to Validity: Threats to internal validity refers to errors inour code and experiment bias. We have double checked our code,
however, there may exist some errors that we did not notice. Also,
to identify the crash releases, we looked for the keywords “crash”,“crashed”, “crashing”, and “crashes”. In such a way, we may increase the number of false positives and false negatives. To reduce
the number of false positives, for each identified crashing release,
we manually checked the commit logs and modified source code.
To reduce the number of false negatives, for these non-crashing releases, we also randomly choose and manually check 30% of thereleases from these non-crashing releases, and we found all of these
releases labeled as non-crashing releases were correctly labeled.
Set and selection bias is another threat to internal validity. To
mitigate this bias, we run the 10-fold cross-validation 100 times,
and present the average performance. To identify crashing releases,
we manually check the commit logs, bug reports, code change, anduser reviews. Due to the manual process, some releases may be
wrongly identified as non-crashing/crashing releases.
Threats to external validity relates to the generalizability of ourresults. We have analyzed 2,638 releases from 10 mobile applications. In the future, we plan to reduce this threat further by analyzing even more releases from additional mobile applications.
Threats to construct validity refers to the suitability of our evaluation measures. We use F1 and AUC scores which is also used bypast studies to evaluate the effectiveness of various software engineering studies [19, 24, 27, 28, 43, 48, 55, 56, 62].

7.

RELATED WORK

Prediction in Software Engineering: There have been a numberof studies that use prediction in software engineering (SE). Kim
et al. proposed the change classification problem, which predicts whether a change is buggy or clean [24]. Jiang et al. proposed
PCC, which built a separate prediction model for each developer
to predict software defects [19]. Bhattacharya et al. proposed a
set of graph-based metrics to predict the number of defects in a
release [3]. Dhaliwal et al. [7] used crash report data to assist in
the bug fixing of Mozilla Firefox bugs. Francese et al. use requirements measures to predict software project and product measures in
the context of Android mobile apps [12]. Yu and Yeung use installand uninstall log in a mobile app store to estimate the value of theapps [64]. Finkelstein et al. extract a set of features from release
notes available in app store, and they find that there is strong correlation between customer rating and the rank of apps in terms ofthe number of downloads [10]. Ferrucci et al. perform a replicatedstudy on the effectiveness of functional and code size measures toapps [9]. Similar to the prior work, we also perform prediction in
SE context. However, our work focuses on a related but different
problem: different from prior defect prediction work, we evaluate
the quality of a mobile application at the release level. In particular,
we predict which mobile releases have a high likelihood of being acrashing release. To the best of our knowledge, our work is the firstto predict crashing mobile app releases.
Mobile Applications: There have been a number of studies in thearea of mobile apps. Martin et al. perform a comprehensive survey
on app store analysis for software engineering [33]. Minelli and
Lanza studied the difference between the development of mobile
application and traditional software systems [36]. They develop a
web-based software analytics platform named SAMOA. Harman
et al. found that there are strong correlations between the rating ofmobile apps and the ranking of app downloads by mining the reviews from App stores [16]. Joorbachi et al. investigated the challenges in mobile application development [20]. They found that
dealing with multiple mobile platforms is one of the most challenging aspects of mobile development. Martin et al. find that therelease context plays a role in whether a release is impactful and thetype of impact it has [31, 32]. Nayebi et al. perform two surveys
to understand the common release strategies used for mobile apps,
the rationale behind them and their perceived impact on users [44].
Martin et al. propose the app sampling problem, and study its effects on sets of user review data [30]. Guerrouj et al. study the impact of app churn on the app rating by analyzing 154 free apps, andthey find that high app churn leads to lower user ratings [13]. Most
related is our prior work, where we performed an empirical study
on the reasons behind low ratings and complaints that were posted
on mobile app reviews [22, 23]. We found that a high percentage
of complaints and bad reviews were posted after new releases. The
work presented in this paper is motivated by the findings of our prior findings, however, our study is orthogonal since our goal is topredict crashing mobile app releases.

8.

CONCLUSION AND FUTURE WORKIn this paper, we perform a study on the prediction of crashingreleases in mobile applications. We first collect 2,638 releases from
10 mobile applications in F-Droid. Then, we extract various factors
from the collected releases. Based on the factors, we propose theusage of a Naive Bayes model to predict crashing releases. To investigate the performance of our proposed approach, we perform
experiments on the 2,638 releases. On average across the 10 apps, our approach can achieve F1 and AUC scores to 0.30 and 0.64,
which improves the random prediction by 50% and 28%, and improves the AUC scores of majority prediction by 30%, respectively.
Our work is one of the first works focusing on the prediction ofcrashing releases. Although the performance of our approach is not
perfect, we hope that our work will inspire other researchers to develop more advanced techniques to identify crashing releases, since
it is clearly a problem that negatively impacts both, mobile users
and developers. In the future, we plan to explore more factors andevaluate our approach with datasets from more mobile applications, develop a better technique that can further improve the prediction
performance, and evaluate the performance of our proposed tool bydeploying it in practice. We also plan to investigate how the integration of bug localization techniques into our technique can help
developers in the fixing process of crashing releases. Moreover,
we plan to perform an empirical study on developers’ perception
on crash release prediction. Recently, Bowes et al. investigated
the usefulness of mutation metrics for fault prediction [4]. In the
future, we plan to investigate if these metrics are useful to predict
crashing releases.
Acknowledgment. This research was partially supported by
JSPS KAKENHI Grant Number 15H05306, the NSFC Program(No.61572426), and National Key Technology R&D Program of
the Ministry of Science and Technology of China under grant
2015BAH17F01.

9.

REFERENCES[1] K. Allix, Q. Jerome, T. F. Bissyande, J. Klein, R. State, and Y. le Traon. A forensic analysis ofandroid malware–how is malware written and how it could be detected? In COMPSAC, 2014.
[2] Android, Apple, Google, Microsoft, AppBrain, BlackBerry, V. sources(WindowsCentral.com), and A. (n.d.). Number of apps available in leading app stores as ofjuly 2015.
[3] P. Bhattacharya, M. Iliofotou, I. Neamtiu, and M. Faloutsos. Graph-based analysis andprediction for software evolution. In ICSE, 2012.
[4] D. Bowes, T. Hall, M. Harman, Y. Jia, F. Sarro, and F. Wu. Mutation-aware fault prediction.
In ISSTA 2016.
[5] L. Breiman. Random forests. Machine learning, 2001.
[6] N. Cliff. Ordinal methods for behavioral data analysis. 2014.
[7] T. Dhaliwal, F. Khomh, and Y. Zou. Classifying field crash reports for fixing bugs: A casestudy of mozilla firefox. In ICSM 2011.
[8] A. Estabrooks and N. Japkowicz. A mixture-of-experts framework for learning from
imbalanced data sets. In Advances in Intelligent Data Analysis. 2001.
[9] F. Ferrucci, C. Gravino, P. Salza, and F. Sarro. Investigating functional and code size
measures for mobile applications: A replicated study. In Product-Focused Software Process
Improvement.
[10] A. Finkelstein, M. Harman, Y. Jia, F. Sarro, and Y. Zhang. Mining app stores: Extracting
technical, business and customer rating information for analysis and prediction. RN, 13:21,
2013.
[11] A. S. Foulkes. Applied statistical genetics with R. 2009.
[12] R. Francese, C. Gravino, M. Risi, G. Scanniello, and G. Tortora. On the use of requirements
measures to predict software project and product measures in the context of android mobile
apps: a preliminary study. In EUROMICRO-SEAA 2015.
[13] L. Guerrouj, S. Azad, and P. C. Rigby. The influence of app churn on app success andstackoverflow discussions. In SANER 2015.
[14] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The weka data
mining software: an update. ACM SIGKDD explorations newsletter, 2009.
[15] J. Han and M. Kamber. Data Mining, Southeast Asia Edition: Concepts and Techniques.
2006.
[16] M. Harman, Y. Jia, and Y. Zhang. App store mining and analysis: Msr for app stores. In MSR,
2012.
[17] A. E. Hassan. Predicting faults using the complexity of code changes. In ICSE 2009.
[18] N. Japkowicz and S. Stephen. The class imbalance problem: A systematic study. Intelligent
data analysis, 2002.
[19] T. Jiang, L. Tan, and S. Kim. Personalized defect prediction. In ASE 2013.
[20] M. E. Joorabchi, A. Mesbah, and P. Kruchten. Real challenges in mobile app development. In
ESEM, 2013.

[21] Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi. A
large-scale empirical study of just-in-time quality assurance. TSE, 2013.
[22] H. Khalid. On identifying user complaints of ios apps. In ICSE, 2013.
[23] H. Khalid, E. Shihab, M. Nagappan, and A. E. Hassan. What do mobile app users complain
about? a study on free ios apps. IEEE Software, 2015.
[24] S. Kim, E. J. Whitehead, and Y. Zhang. Classifying software changes: Clean or buggy? TSE.
[25] S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller. Predicting faults from cachedhistory. In ICSE, 2007.
[26] D. E. Krutz, M. Mirakhorli, S. A. Malachowsky, A. Ruiz, J. Peterson, A. Filipski, and
J. Smith. A dataset of open-source android applications. In MSR.
[27] A. Lamkanfi, S. Demeyer, E. Giger, and B. Goethals. Predicting the severity of a reported
bug. In MSR, 2010.
[28] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch. Benchmarking classification models forsoftware defect prediction: A proposed framework and novel findings. TSE.
[29] A. Machiry, R. Tahiliani, and M. Naik. Dynodroid: An input generation system for android
apps. In FSE, 2013.
[30] W. Martin, M. Harman, Y. Jia, F. Sarro, and Y. Zhang. The app sampling problem for app
store mining. In MSR 2015.
[31] W. Martin, F. Sarro, and M. Harman. Causal impact analysis for app releases in google play.
In FSE 2016.
[32] W. Martin, F. Sarro, and M. Harman. Causal impact analysis applied to app releases in google
play and windows phone store. RN, 15:07, 2015.
[33] W. Martin, F. Sarro, Y. Jia, Y. Zhang, and M. Harman. A survey of app store analysis forsoftware engineering. RN, 16:02, 2016.
[34] A. McCallum, K. Nigam, et al. A comparison of event models for naive bayes textclassification. In AAAI-98 workshop.
[35] T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener. Defect prediction fromstatic code features: current results, limitations, new approaches. Automated Software
Engineering, 2010.
[36] R. Minelli and M. Lanza. Software analytics for mobile applications–insights &amp; lessons
learned. In CSMR, 2013.
[37] A. Mockus. Organizational volatility and its effects on software defects. In FSE, 2010.
[38] A. Mockus and D. M. Weiss. Predicting risk of software changes. Bell Labs Technical
Journal, 2000.
[39] R. Moser, W. Pedrycz, and G. Succi. A comparative analysis of the efficiency of change
metrics and static code attributes for defect prediction. In ICSE, 2008.
[40] H. Muccini, A. Di Francesco, and P. Esposito. Software testing of mobile applications:
Challenges and future research directions. In AST, 2012.
[41] N. Nagappan and T. Ball. Use of relative code churn measures to predict system defectdensity. In ICSE, 2005.
[42] N. Nagappan, T. Ball, and A. Zeller. Mining metrics to predict component failures. In ICSE,
2006.
[43] J. Nam, S. J. Pan, and S. Kim. Transfer defect learning. In ICSE 2013.
[44] M. Nayebi, B. Adams, and G. Ruhe. Release practices in mobile apps? users and developers
perception. In SANER 2016.
[45] F. Rahman, D. Posnett, A. Hindle, E. Barr, and P. Devanbu. Bugcache for inspections: hit or
miss? In ESEC/FSE, 2011.
[46] J. D. Rennie, L. Shih, J. Teevan, D. R. Karger, et al. Tackling the poor assumptions of naive
bayes text classifiers. In ICML, 2003.
[47] B. Robinson and P. Francis. Improving industrial adoption of software engineering research: acomparison of open and closed source software. In ESEM 2010, ESEM ’10.
[48] D. Romano and M. Pinzger. Using source code metrics to predict change-prone javainterfaces. In ICSM, 2011.
[49] C. Rosen, B. Grawi, and E. Shihab. Commit guru: Analytics and risk prediction of softwarecommits. In ESEC/FSE 2015.
[50] C. Rosen and E. Shihab. What are mobile developers asking about? a large scale study using
stack overflow. EMSE.
[51] E. Shihab, A. E. Hassan, B. Adams, and Z. M. Jiang. An industrial study on the risk ofsoftware changes. In FSE, 2012.
[52] J. Śliwerski, T. Zimmermann, and A. Zeller. When do changes induce fixes? SIGSOFT Softw.
Eng. Notes, 2005.
[53] J. Su, H. Zhang, C. X. Ling, and S. Matwin. Discriminative parameter learning for bayesian
networks. In ICML, 2008.
[54] R. Subramanyam and M. S. Krishnan. Empirical analysis of ck metrics for object-oriented
design complexity: Implications for software defects. TSE, 2003.
[55] F. Thung, D. Lo, M. H. Osman, and M. R. Chaudron. Condensing class diagrams byanalyzing design and network metrics using optimistic classification. In ICPC, 2014.
[56] Y. Tian, J. Lawall, and D. Lo. Identifying linux bug fixing patches. In ICSE, 2012.
[57] Y. Tian, D. Lo, and C. Sun. Drone: Predicting priority of reported bugs by multi-factor
analysis. In ICSM, 2013.
[58] B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano. On the relative value of cross-company
and within-company data for defect prediction. EMSE, 2009.
[59] H. Valdivia and E. Shihab. Characterizing and predicting blocking bugs in open source
projects. In MSR, 2014.
[60] H. Valdivia Garcia and E. Shihab. Characterizing and predicting blocking bugs in open source
projects. In MSR, 2014.
[61] F. Wilcoxon. Some rapid approximate statistical procedures. Annals of the New York Academyof Sciences, 52(6):808–814, 1950.
[62] R. Wu, H. Zhang, S. Kim, and S.-C. Cheung. Relink: recovering links between bugs andchanges. In ESEC/FSE, 2011.
[63] X. Xia, D. Lo, E. Shihab, X. Wang, and X. Yang. Elblocker: Predicting blocking bugs withensemble imbalance learning. Information and Software Technology, 61:93–106, 2015.
[64] P. Yu and C.-m. Au Yeung. App mining: finding the real value of mobile applications. In
WWW, 2014.
[65] H. Zimmermann. Fuzzy Set Theory and Its Applications Second, Revised Edition. 1992.