Predicting numeric ratings for Google apps using text features and ensemble learning

ETRI JournalVolume 43, Issue 1 p. 95-108 ORIGINAL ARTICLEFree Access Predicting numeric ratings for Google apps using text features and ensemble learning Muhammad Umer, orcid.org/0000-0002-6015-9326 Department of Computer Science, Khawaja Freed University, Punjab, PakistanSearch for more papers by this authorImran Ashraf, Department of Information and Communication Engineering, Yeungnam Univeristy, Gyeongsan, Rep. of KoreaSearch for more papers by this authorArif Mehmood, Department of Computer Science and Information Technology, The Islamia University of Bahawalpur, Punjab, PakistanSearch for more papers by this authorSaleem Ullah, Corresponding Author saleem.ullah@kfueit.edu.pk orcid.org/0000-0003-3747-1263 Department of Computer Science, Khawaja Freed University, Punjab, Pakistan Correspondence Gyu Sang Choi, Department of Information and Communication Engineering, Yeungnam University, Gyeongsan, Korea. Email: castchoi@ynu.ac.kr Saleem Ullah, Department of Computer Science, Khawaja Freed University, Punjab, Paksitan. Email: saleem.ullah@kfueit.edu.pkSearch for more papers by this authorGyu Sang Choi, Corresponding Author castchoi@ynu.ac.kr Department of Information and Communication Engineering, Yeungnam Univeristy, Gyeongsan, Rep. of Korea Correspondence Gyu Sang Choi, Department of Information and Communication Engineering, Yeungnam University, Gyeongsan, Korea. Email: castchoi@ynu.ac.kr Saleem Ullah, Department of Computer Science, Khawaja Freed University, Punjab, Paksitan. Email: saleem.ullah@kfueit.edu.pkSearch for more papers by this author Muhammad Umer, orcid.org/0000-0002-6015-9326 Department of Computer Science, Khawaja Freed University, Punjab, PakistanSearch for more papers by this authorImran Ashraf, Department of Information and Communication Engineering, Yeungnam Univeristy, Gyeongsan, Rep. of KoreaSearch for more papers by this authorArif Mehmood, Department of Computer Science and Information Technology, The Islamia University of Bahawalpur, Punjab, PakistanSearch for more papers by this authorSaleem Ullah, Corresponding Author saleem.ullah@kfueit.edu.pk orcid.org/0000-0003-3747-1263 Department of Computer Science, Khawaja Freed University, Punjab, Pakistan Correspondence Gyu Sang Choi, Department of Information and Communication Engineering, Yeungnam University, Gyeongsan, Korea. Email: castchoi@ynu.ac.kr Saleem Ullah, Department of Computer Science, Khawaja Freed University, Punjab, Paksitan. Email: saleem.ullah@kfueit.edu.pkSearch for more papers by this authorGyu Sang Choi, Corresponding Author castchoi@ynu.ac.kr Department of Information and Communication Engineering, Yeungnam Univeristy, Gyeongsan, Rep. of Korea Correspondence Gyu Sang Choi, Department of Information and Communication Engineering, Yeungnam University, Gyeongsan, Korea. Email: castchoi@ynu.ac.kr Saleem Ullah, Department of Computer Science, Khawaja Freed University, Punjab, Paksitan. Email: saleem.ullah@kfueit.edu.pkSearch for more papers by this author First published: 06 July 2020 https://doi.org/10.4218/etrij.2019-0443Citations: 3 Muhammad Umer and Imran Ashraf contributed equally. Funding information: This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1A2C1006159), and MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2019-2016-0-00313) supervised by the IITP (Institute for Information & Communications Technology Promotion). AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Application (app) ratings are feedback provided voluntarily by users and serve as important evaluation criteria for apps. However, these ratings can often be biased owing to insufficient or missing votes. Additionally, significant differences have been observed between numeric ratings and user reviews. This study aims to predict the numeric ratings of Google apps using machine learning classifiers. It exploits numeric app ratings provided by users as training data and returns authentic mobile app ratings by analyzing user reviews. An ensemble learning model is proposed for this purpose that considers term frequency/inverse document frequency (TF/IDF) features. Three TF/IDF features, including unigrams, bigrams, and trigrams, were used. The dataset was scraped from the Google Play store, extracting data from 14 different app categories. Biased and unbiased user ratings were discriminated using TextBlob analysis to formulate the ground truth, from which the classifier prediction accuracy was then evaluated. The results demonstrate the high potential for machine learning-based classifiers to predict authentic numeric ratings based on actual user reviews. 1 INTRODUCTION Opinion mining is the process of identifying and detecting subjective information using natural language processing, text analysis, and computational linguistics. It is a subfield of text mining, which aims at building systems that identify and extract opinions within a text. Opinions play a significant role in decision-making in daily life. The ubiquity of smartphones increasingly impacts our work- and lifestyles. Hundreds of thousands of applications (“apps”) provide a multitude of everyday services encompassing medical care, fitness, beauty, surveillance, sports, etc. As of March 2019, a total of 2.6 million apps are available on the Google Play store [1], as shown in Figure 1A. Millions of users can download, use, and evaluate these apps and provide feedback in the form of reviews and numeric ratings. Currently, 205.4 billion apps are downloaded annually and this number is expected to reach 352.9 billion by 2021 [2], as shown in Figure 1B. User reviews and numeric ratings are the leading guiding factors for other users considering whether to adopt particular apps. Research [3] has demonstrated the strong influence of user reviews and numeric ratings in spreading the use of mobile apps generally. It was shown that consumers can prefer to buy a 5-star- rather than a 4-star-rated product, even at an additional cost of 20%–99%. The ability to share ratings, bug reports, and reviews increases the scope of engagement of users and app developers alike [4]. FIGURE 1Open in figure viewerPowerPoint Evolution of apps on the Google Play store [1, 2]: (A) Total apps on the Google Play Store, (B) Total apps downloaded from the Google Play Store User feedback is provided in two forms, text review and numeric ratings, as illustrated in Figure 2. A text review, on the one hand, may contain positive or negative comments submitted by a user about a specific app or policy. These data can be highly useful for performing marketing analysis, managing public relations, conducting product reviews, net promoter scoring, giving product feedback, delivering customer service, and so on [5]. FIGURE 2Open in figure viewerPowerPoint The Google Play store apps review A numeric rating, on the other hand, is a quantitative value given by a user, ranging from 1 to 5. It is an aggregated rating, typically defined by an app user selecting one or more star icons. A high numeric rating motivates more potential users to download the app. Given the importance of reviews and app ratings, the possibility of biased and fake reviews is a substantial concern. Currently, there is no standard mechanism for validating the authenticity of numeric user ratings. This causes inconvenience and uncertainty for end users wishing to select the most suitable mobile apps. Predominantly, users tend to select an app based only on the first comments and reviews they read. Typically, time-pressured users feel they do not have the leisure to read entire reviews, which results in their selecting the wrong app. Although several approaches to address these problems have been proposed [6, 7], they focus essentially on identifying only a polarity inherent in user reviews. Polarity and subjectivity are defined in terms of three descriptors: positive, negative, and neutral. These approaches focus on reviews exclusively without considering actual app ratings, thereby ignoring the problem of mismatched numeric ratings. Our experiments show that displayed numeric ratings and user reviews are often mismatched, resulting in contradictory evaluations. This proposes an approach designed to overcome this shortcoming by predicting the numeric rating for a mobile app based on user reviews. A machine learning-based ensemble classifier is used for this purpose. The dataset is first preprocessed by applying a case transformation, removing numbers, removing stop words, tokenization, and stemming. To balance the dataset and avoid training classifiers disproportionately on 5-star-rated instances, we extracted 25 000 records associated with each target rating to train our ensemble learning classifiers on all the text features amenable to rating. Hence, the new dataset contained 125 000 instances. We then applied the vector-space modeling (VSM) techniques of term frequency/inverse document frequency (TF/IDF), considering unigrams, bigrams, and trigrams and TF, to the preprocessed data. Features generated by these techniques were then passed to ensemble learning models. The key contributions of this research can be summarized as follows: Machine learning algorithms, including the random forest (RF), the gradient boosting classifier (GBM), the extreme gradient boosting classifier (XGB), the AdaBoost classifier (AB), and the extra tree classifier (ET) were applied to predict numeric ratings. The classifier accuracy was analyzed from the following two perspectives: (a) using the textual features derived from the reviews alone, and (b) using both the textual features and the emoticons available in the reviews. For validation, famous apps belonging to each category were used to compare the numeric ratings predicted by ensemble learning with user's actual ratings. The rest of the paper is organized as follows. Section 2 reviews prior-related research. Section 3 introduces the background information on the ensemble learning classifiers used in this study. Section 4 presents our proposed approach and the experimental dataset. The results are discussed in Section 5 and concluding remarks are given in Section 6. 2 RELATED WORK There has been a constant growth in the public and private information stored within the internet. This includes textual data expressing people's opinions on review sites, forums, blogs, and other social media platforms. Review-based prediction systems allow this unstructured information to be automatically transformed into structured data reflecting public opinion. These structured data can be used subsequently as a measure of users' sentiments about specific applications, products, services, and brands. They can hence provide important information for product and services refinement. This kind of sentiment analysis was conducted in the following studies. Kumari and others [8] and other researchers [5, 9, 10] used the naïve Bayes (NB) classifier to classify opinions as positive, negative, or neutral. Wang and others [11] argued that a rating is not entirely determined by a review content. For example, a user may well intend to give a positive review by employing positive words, and yet issue a comparatively lower rating. Dave and others [12] proposed a method for extracting the polarity in user reviews of products, expressed as poor, mixed, or good. The classifier used was NB. According to Pang et al., although machine learning approaches perform much better for traditional topic-based categorization, they are less successful for sentiment analysis [13]. Information-extraction technologies have also been explored to identify and organize opinions contained in text. For example, some authors [14] proposed a scheme for annotating a low-level representation of opinions within a text. Additionally, they described an opinion-oriented “scenario template” that summarizes the opinions expressed in a document. This approach is helpful for tasks that involve posing question from multiple perspectives. Other authors [15] suggested adopting a statistical analysis based on a spin model, to extract the semantic orientations of words. Mean-field approximations were used to compute the approximate probability in the spin model. Semantic orientations are then evaluated as desirable or undesirable. A smaller number of seed words for the proposed model produce highly accurate semantic orientations based on the English lexicon. Various sentiment analysis methods have been performed to summarize the ensembles of comments and reviews [16]. These methods use mathematical and statistical methods (especially involving Gaussian distributions) to overcome the problems encountered in sentiment analysis. Although these authors proposed a model, it was not implemented. A recent study [17] investigated the application of a machine learning algorithm to a dataset covering, for example, the app category, the numbers of reviews and downloads, the size, type, and Android version of an app, and the content rating, to predict a Google app ranking. Decision trees, linear regression, logistic regression, support-vector machine, NB classifiers, k-means clustering, k-nearest neighbors, and artificial neural networks were studied for that purpose. App ratings have been predicted based on the features provided for app [18, 19]. Experiments were performed on the BlackBerry World and Samsung Android stores to collect the raw features provided for the apps, including their price, rank of downloads, ratings, and textual descriptions. The features were then encoded into a numerical vector to be used in case-based reasoning and to predict the app rating. In contrast to the above-cited studies, other authors [20] investigated the nature of sentiments expressed in Google app reviews. Their study measured opinions and sentiments represented in user reviews through a variety of emojis expressing, for example, negativity, positivity, anger, or excitement. It evaluated whether those sentiments are informative for the purpose of app development and refinement. However, the above studies are unsatisfactory in various respects and are unsuitable for predicting numeric ratings of Google apps. First, text-mining techniques are ineffective when applied to app reviews, as it has Unicode supported language with a limited number of words. Second, those studies are based either on rating predictions made using inherent app features or on external features (eg, price, bug report, etc.). None of those studies investigated the possible discrepancies between users' numeric ratings and reviews. To our knowledge, this study is the first to investigate such discrepancies and to base numeric-rating predictions for Google apps on users' reviews. 3 ENSEMBLE LEARNING METHODS USED FOR PREDICTION Our approach exploits ensemble learning classifiers to predict numeric ratings based on users' reviews of mobile apps. We evaluated the performance of various ensemble learning classifiers, discussed briefly below. Models were implemented using Scikit learn [21, 22] in Python. Since ensemble learning methods are meta-algorithms that combine several machine learning techniques into one predictive model, they help to decrease the variance (bagging) and bias (boosting) and improve predictions (stacking) [23]. There is a copious literature on the lexicon- and machine learning-based classification of unstructured and structured data. However, machine learning-based classifiers are preferred over other methods on account of their superior performance. For example, some authors [24] compared the performance of 10 models (five each from lexicon- and machine learning-based classifiers) in terms of their classification efficacy. They concluded that RF and NB classifiers are substantially better at correctly uncovering human intuition. In contrast, lexicon-based approaches perform worse than machine learning. These findings were corroborated by other authors who concluded that machine learning-based classifiers outperform lexicon-based approaches [25, 26]. In light of other studies [24-26], the present study uses machine learning-based classifiers to predict Google app ratings. RF provides an alternative to decision trees. It reduces the variance by generating several trees; however, the results are not easily interpretable. It is used for classification and regression. It involves an ensemble of decision trees in which results are aggregated into one final result. RF reduces the variance in two ways: first by training on different data samples and second by using a random subset of features [27]. AdaBoost (AB), a short form for adaptive boosting, is the first boosting algorithm to outperform other boosting algorithms in classification tasks in terms of accuracy. AdaBoost is adaptive in the sense that weak learners are adapted by considering earlier misclassifications by previous classifiers [28]. Any individual learner can be weak. However, when considering a complete learned model, AdaBoost achieves a high accuracy. It is also used for finding important features by calculating a feature importance score. Gradient Boosting (GB) is an ensemble learning technique used for regression and classification problems. It produces a prediction model consisting of an ensemble of weak prediction models, typically in the form of decision trees [29]. Boosting is a method for converting weak learners into strong learners. In boosting, each new tree is fitted to a modified version of the original dataset. The good performance of GB stems from its use of gradients in the loss function, which measures how efficiently the model coefficients fit the underlying data. The precise definition of the loss function depends on which quantity, feature, or property is to be optimized. Extreme gradient boosting (XGBoost) is a boosting algorithm with much appeal among data scientists for its execution speed and performance. Its good performance stems from its implementation, which is composed of three components, namely gradient boosting, stochastic gradient boosting, and regularized gradient boosting. It is used for both classification and regression problems. XGBoost attracted attention when it earned success to several Kaggle competition winners [30]. Feature importance scores are also calculated, as with other boosting and bagging classifiers. The extra-trees (ET) classifier implements a meta-estimator that fits a number of randomized decision trees (ET) on various subsamples of a dataset, performs averaging to improve the predictive accuracy, and controls overfitting [31]. Unlike RF, the entire sample is used by the ET classifier at each step and decision boundaries are picked randomly, rather than by selecting the best one. It is also known as “extremely randomized tree.” 4 PROPOSED APPROACH FOR EVALUATING APPLICATION RATINGS This section describes the proposed approach, its modules, and the dataset used in the experiment. The architecture of the proposed approach for predicting numeric ratings is outlined in Figure 3. It involves several subtasks, described separately below. FIGURE 3Open in figure viewerPowerPoint Architecture of the proposed approach 4.1 Dataset The Google apps dataset was scraped from the Google Play store using the BeautifulSoup web scraper. The data were scraped for applications released no later than 2014 to ensure a minimum time span of 5 years. The following criteria were applied: The app must be 5 years old at least. The app must have at least 4000 posted reviews. The dataset contains a total of 502 658 records and includes the following attributes, described in Table 1: App_ category, App_ name, App_ id, App_ review, and App_ rating. Although other features and related meta-data can be scraped from the Google Play store in principle, this study focused on analyzing the discrepancy between user reviews and app-rating predictions based on the features extracted from the reviews. We therefore neglected other meta-data that were irrelevant. TABLE 1. Description of attributes in the used dataset Attribute Description App_ category App category in the Google Play store App_ name Actual app name in the Google Play store App_ id Unique app_ id assigned in the Google Play store for identification App_ review Review given by each individual user for a specific app App_ rating Rating given for the app by an individual user. The scraped data consist of 14 different categories of mobile apps. The primary motivation for sampling over multiple categories was to cover the broad range of reviews available. Each category (eg, sports, news, entertainment, health, and business) contains different types of textual reviews displaying a variety of expressions and words. Our purpose was therefore to evaluate the performance of the classifiers by applying them to diverse reviews, rather than considering only a few categories, to properly assess the classification accuracy. Table 2 shows a sample of reviews obtained from the photography, health and fitness, finance, weather, and medical categories. TABLE 2. Actual reviews from various app categories Category Reviews Photography 'I TOOK SOME PICTURES AND IT HAS A GOOD SNAP I HAVEN'T TRIED THE VIDEO RECORDER YET BUT SURE IT'S GOOD AS THE CAMERA KEEP ON BANGING' Health & Fitness 'I loved this App,Really All the Exercises are Effective and Awesome to do…Also The Different features like Time, Day Tracker Helps a lot… I just have one query… Will These Exercises affect my height ? I am worried about my Height…I want it to increase.' Finance 'ATM withdrawal feature via QR codes is broken on Google Pixel due to the app using a wrong aspect ratio with the camera – a square QR code is captured as 2:1, making it not recognizable. Please fix, if you could. Thanks.' Weather 'Awesome app. Lots of information. Thunderstorms, live radar, hurricane Tracking. 5 day forecast. Everything you need to know about the weather.' Medical 'I am a Physiotherapist and I need to make my clients understand about the musculoskeletal system and the part that is involved and I always wished if such things (apps) will ever be made. Thank to the creator of this app.' Figure 4 shows the names and relative size of all the categories in the dataset. Each app in the scraped database has at least 4000 reviews. The App_ rating and App_ review data from individual users were analyzed. The data were scraped using the BeautifulSoup (BS) web scraper, as depicted in the flow chart in Figure 5. FIGURE 4Open in figure viewerPowerPoint Count of each category in the dataset FIGURE 5Open in figure viewerPowerPoint Flow diagram for data scraping Data visualization is essential for understanding the dataset by identifying patterns, trends, and correlations that might go undetected in text-based data. In several datasets, such connections can be recognized conveniently through data visualization. Therefore, we visualized the dataset to display the numeric ratings given to mobile apps. Figure 6 identifies 5 as the most frequent app user rating. However, the possibility of biased or even fake ratings given by anonymous users is a major concern. We therefore considered the frequency of numeric ratings in each category. Figure 7 shows that mobile apps in the casual category always score higher ratings and are most likely to display a 5-star rating from app users. In contrast, developers of Video Players & Editors tend to receive the lowest numeric ratings. FIGURE 6Open in figure viewerPowerPoint Frequency of user ratings FIGURE 7Open in figure viewerPowerPoint User ratings by category 4.2 Date preprocessing The dataset collected from the Google Play store is semi-structured or unstructured and contains significant superfluous data (defined as not contributing significantly to the prediction process). Since large datasets require longer training times, and because “stop words” reduce the prediction accuracy, text preprocessing is therefore required to overcome this limitation. Preprocessing involves various tasks including stemming, lowercase conversion, punctuation, and excluding terms which do not carry higher weight in context to the text. Research has shown that text preprocessing plays a significant role in improving the prediction accuracy [32]. The data must therefore be normalized before starting the training process using the proposed approach. To better understand text preprocessing, consider the example of a review for the mobile app Seven minute workout challenge. A series of preprocessing steps and their output are shown in Table 3. After the completion of preprocessing, the ensemble learning classifiers can be applied to the processed dataset. TABLE 3. Attributes of the used dataset Preprocessing steps Output Actual text '\ u003c/\ u003e' I honestly preferred the older version, but the new one isn't to bad. The demonstrator keeps glitching out, and the next exercise keeps popping up while I'm doing a different one.’ Transform to lower case '\ u003c/\ u003e' i honestly preferred the older version, but the new one isnt to bad. the demonstrator keeps glitching out, and the next exercise keeps popping up while im doing a different one.’ Remov HTML tags i honestly preferred the older version, but the new one isnt to bad. the demonstrator keeps glitching out, and the next exercise keeps popping up while im doing a different one.’ Remove stop words & tokenization ['honestly', 'preferred', 'older', 'version', ',', 'new', 'one', 'isnt', 'bad', '.', 'demonstrator', 'keeps', 'glitching', ',', 'next', 'exercise', 'keeps', 'popping', 'im', 'different', 'one', '.'] Stemming ‘honestly prefer older version, new one isn't bad. demonstrate keep glitch, next exercise keep pop im differ one.’ 4.3 Feature selection Training a supervised machine learning algorithm requires textual documents to be represented in vectorial form. For this purpose, textual data must be converted into numbers without losing information. This transformation can be achieved using several techniques, for example, Bag-Of-Words (BOW), which assigns a number to each word. However, character-length limitations in reviews can reduced the efficiency of BOW. Insufficient word occurrences can also limit the accuracy of BOW-based approaches [33]. We therefore exploit the term frequency (TF) to perform the transformation [34]. This involves counting the words in a document to produce a matrix displaying the total number of occurrences of each word in the document. As an example of TF, consider a sample review for the app My Talking Tom. The original review is: It is great, my kids love to play it. Thank you for having this game on game store. After preprocessing, the review becomes great kid love play. thank game game store. The matrix obtained by applying TF is shown in Table 4. TABLE 4. Features obtained by applying TF Game Great Kid love Play Store Thank 2 1 1 1 1 1 1 We also used TF/IDF (for unigrams, bigrams, and trigrams) to perform feature selection in our proposed approach [35]. TF/IDF assigns a lesser weighting to highly common words (occurring in almost all the documents), but gives a stronger weighting to the words that appear more specifically in a subset of documents. Commonly occurring words are thus assigned a lower weight, while some rare words occurring more specifically in a particular document are considered more important. The outcome of TF/IDF on the above example is shown in Table 5. TABLE 5. Features obtained by applying TF/IDF Game Great Kid love Play Store Thank 0.63 0.31 0.31 0.31 0.31 0.31 0.31 4.4 Accuracy measures We used different parameters for performance evaluation. Four basic notions are important for understanding the different accuracy parameters of a classifier [36, 37]. True Positives (TP): positive predictions about a class which are correctly predicted by the classifier. True Negatives (TN): negative predictions about a class which are correctly labeled by the classifier. False Positives (FP): negative instances of a class which are incorrectly labeled as positive by the classifier. False Negatives (FN): positive instances of a class which are incorrectly labeled as negative by the classifier. Accuracy is an evaluation parameter widely used in classification models. In terms of TP, TN, FP, and FN, defined above, it is calculated as follows: (1) Precision, in contrast, specifies the percentage of all instances of a class that are correctly labeled as positive, that is, (2) Recall is often considered as a measure of completeness and represents the proportion of true positive instances of a class that are labeled as such, that is, (3) The F score, which takes values between 0 and 1, considers both the precision and the recall. It represents the average effect of the precision and recall, calculated as (4) 5 EXPERIMENT AND RESULTS This section details the experiment conducted herein, and discusses the results. To predict authentic numeric rating using ensemble learnin