# Investigating the Effect of User Reviews on Mobile Apps: The Role of Customer Led Innovation

Miriam Erne, Zhiying Jiang, Vanessa Liu

► To cite this version:

Miriam Erne, Zhiying Jiang, Vanessa Liu. Investigating the Effect of User Reviews on Mobile Apps: The Role of Customer Led Innovation. International Working Conference on Transfer and Diffusion of IT (TDIT), Dec 2020, Tiruchirappalli, India. pp.193-200, 10.1007/978-3-030-64849-7_18. hal-03701795

HAL Id: hal-03701795

https://inria.hal.science/hal-03701795

Submitted on 22 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Investigating the Effect of User Reviews on

Mobile Apps: The Role of Customer Led Innovation

Miriam Erne¹, Zhiying Jiang² and Vanessa Liu²

¹ Erasmus University Rotterdam, Burgemeester Oudlaan 50, 3062 PA Rotterdam, Netherlands

² Singapore University of Social Sciences, 463 Clementi Rd, Singapore 599494

vanessaliusw@suss.edu.sg

**Abstract.** User involvement has been made easy and common in the context of mobile applications (apps), where user reviews were often collected to enlighten apps developers on novel features. However, users might not always possess the required technical expertise to make commercially feasible suggestions. The value of user reviews also varied due to their unmanageable volume and content irrelevance. In this study, over 40,000 user reviews with 50 apps were analyzed to empirically examine the association between customer led innovation and the revenues from the apps. Our findings indicated that customer led innovation alone did not lead to significant changes in revenues. Its impact was only significant if the developers responded to the user reviews faster. These results contributed to the user involvement literature by highlighting the importance of the moderating effect of developer responsiveness. Apps developers could also benefit from our empirical evidence that proved the value of user involvement that enhanced innovativeness.

**Keywords:** mobile apps, user involvement, customer led innovation, user reviews

# 1 Introduction

Nowadays application distribution platforms such as Apple App Store and Google Play provide millions of different mobile applications (apps) to users. As of the fourth quarter of 2019, there were around 2.57 million apps for android users and 1.84 million apps for App Store users available [22]. Survival in such a “hyper-competitive” mobile market was challenging to apps developers [7]. Unwanted or unpopular apps could be phased out very shortly after launch, resulting in a waste of development cost and effort. To sustain competitiveness, it is therefore becoming increasingly important for apps developers to launch novel features that meet customer needs (e.g., see [5][17][18]).

One key channel for customers to voice out their feedback is through user reviews. Indeed, most apps actively elicit customer comments as they are enlightening to the apps developers in terms of novel features. As user needs vary significantly and the usage of the apps could differ across contexts, users may be a good source of creative ideas for development of innovative functionalities.

Despite its potential usefulness for performance enhancement, screening through user reviews could be challenging. For instance, online gurus like Facebook could generate as high as at least 2,000 user reviews per day [5]. The aspects covered in the reviews could be highly diverse, ranging from complaints about the price of the apps to the frequency of advertisements. Manual processing of these reviews is simply impossible. Management of user reviews could be overwhelming and costly to apps developers. Tools have accordingly been developed to enable automated categorization and mining of customer reviews [17][18]. However, following up on user reviews remains highly time- and money-consuming. Is it really worth the resources to act upon the user reviews? Considering not all users are technically knowledgeable about apps development, could their involvement really offer constructive and commercially feasible suggestions for apps improvement?

User involvement is only appropriate if certain involvement roles and development conditions are fulfilled [12]. These conditions include, who should be involved, which type of software with which the users should be involved, and in which stage (i.e., when) of the software development the users should be involved. User involvement could be totally undesirable when technical expertise is needed. While the potential value of user feedback is not deniable, it may not always be economically justified for developers to translate user feedback into actual software features [12].

Our study therefore aims to empirically investigate the impact of user involvement (in the form of user reviews) on apps performance. Most prior researchers focused on the development of analytical tools for categorization of user reviews (e.g., [17][18]), seldom questioning the actual benefits of the ideas from the user on apps development, presuming that users could always provide useful feedback. In this study, we categorized and analyzed over 40,000 user reviews associated with about 50 apps to verify the impact on user involvement on
apps performance. Specifically, we conceptualized user reviews with innovative suggestions as “customer led innovation” and examined its impact on revenues of apps. We also took into consideration the time taken for apps developers to respond to the user reviews. The value of the innovativeness of user inputs may depreciate over time as other competitors might have already launched similar features onto the market. We hence included developers’ responsiveness as a moderator on the relationship between customer led innovation and apps performance. In other words, the effect of user reviews on apps performance would be significantly greater if the developers were more responsive and attend to the user feedback faster.

The remainder of this paper is structured as follows: first, we will explain the conceptual framework and the related past studies. The research methodology and the data analysis procedure will then be presented. Finally, the findings will be discussed and the theoretical and managerial implications will be drawn.

## 2 Conceptual Framework

Our research model is presented in Figure 1.

<figure><img src="image_2.png" /><figcaption>Figure 1. The Research Model</figcaption></figure>

### 2.1 User Involvement

The notion of user involvement was well documented in the literature, referring to the level of personal relevance and importance attached by users to the system [2]. In broad terms, it is defined as “direct contact with users”[15]. Recently, it was observed that customers had become more and more involved in the product development [21]. User involvement was essential and indispensable for system/ software developers as it helped to collect more accurate user requirements and enable quality improvement, resulting in better fulfillment of user needs and higher user satisfaction [13][16]. User involvement was therefore recognized by previous researchers as beneficial to the improvement of quality and performance [4]. Terms such as co-creation or co-design had emerged to describe the collaboration between developers and users. Other terms included quality function deployment (QFD), user-oriented product development, concept testing, Beta testing, consumer idealized design, lead user method and participatory ergonomics [13]. In the collaborative process, users may assume the roles of providers of information, commentators or objects for observations.

In the context of mobile apps, users and apps developers may exchange ideas on shared platforms such as the App stores. Users could submit their desirable new features or functionalities [14][20]. Complaints from users on lack of certain features could shed light on potential new apps development [3]. However, the number of user reviews received could be immeasurable and unmanageable. More importantly, not all feedback is useful. Almost 65% of apps reviews were found to be noisy and irrelevant [5]. Some suggestions might be solely emotional and commercially infeasible for implementation.

Many tools were therefore developed to aid the search, screening, and extraction of useful information from user reviews. A review of the current literature showed that different tools were built with different mining objectives. Examples included MARK (Mining and Analyzing Reviews by Keywords) [24], MARA (Mobile App Review Analyzer) [11], ALERTme [10], and AR-Miner (App Review Miner) [5]. These tools made use of techniques like natural language processing, topic modeling, clustering and machine learning algorithms to search, classify, extract, group and rank user reviews based on pre-defined keywords or categories.
## 2.2 Customer Led Innovation

User reviews, if carefully and properly screened and processed, could be vital to innovativeness of apps development. For example, a user might point out interesting and novel features that could be added for iPhone users. With many varieties of smartphones available and varied user profiles, it was difficult for apps developers to consider all possible new features. User reviews could be a good source to identify creative solutions. Though some users may be tech-non-savvy, the imaginativeness may never be foreseen in the development process. Their feedback could help developers to visualize innovative features of the apps. We therefore conceptualize user reviews with new feature requests as customer led innovation. It denotes requests from users on new features to be added to the apps or new apps development. Customer led innovation offer insights to developers to add novel features, resulting in greater efficiency of development and higher user satisfaction [16]. Accordingly, we hypothesize that:

H1: Customer led innovation has a significant and positive impact on apps performance.

## 2.3 Developer Responsiveness to User Reviews

The time taken by developers to respond to user reviews on apps innovation may matter [23]. After a user submitted his/her feedback, he/she may tend to expect the developer to address the suggestion quickly. If the developer response is slow, other competing apps in the market might have already introduced the new feature and the degree of novelty to users would be diminished. Conversely, users may tend to be more positive about the apps if their ideas were implemented promptly. The new features would also be perceived as more novel with their first time to market. Developers should, however, be cautioned about the frequency of apps dates. Frequent issuance of updates upon novel features may actually cause disruptions to users. In general, reasonable responsiveness to creative solutions should lead to better quality and performance of apps [1]. The shorter the time taken to respond to user reviews, the greater the effect is the reviews of new features on apps performance. Accordingly, we hypothesize that:

H2: Developer responsiveness significantly and negatively moderates the relationship between customer led innovation and apps performance.

# 3 Research Methodology

## 3.1 Research Context

The data was collected through App Annie, a business intelligence company. It stored data on a collection of health and fitness apps, including the apps user reviews and revenues generated from each app. In our study, only apps that had been active for at least one year were included in the sampling. Active apps should provide more valid results as it was common in the mobile apps industry that numerous apps could have been removed before their official launch. A total of 50 apps were selected for our analysis as their revenue constituted almost 75% of the total revenue in the health and fitness apps market. There were 189,527 user reviews available for these selected apps.

In order to measure the effect of user reviews on apps performance, a specific research time frame was defined. Only reviews posted after the second last updated version and before the latest version of the apps were included in our samples. This enabled us to examine whether the user reviews led to development of new features in the resultant update of the apps. The final sample consisted of a total of 40,619 user reviews, representing 21.4% of the total reviews associated with the selected apps.

## 3.2 Measurement

### Customer led innovation

User reviews were used as proxies for innovations suggested by customers. A subtraction and categorization process were conducted to identify the reviews that specifically pertained to innovation suggestions.

First, generic reviews were subtracted to isolate the specific reviews [5]. Generic reviews were noisy and irrelevant reviews that did not provide any information on creative solutions. Examples of such review were “by far the best app on meditation!” and “I love this app and have done since the moment I started using it. Potentially
helped me get through a period of anxiety...". Specific reviews, on the other hand, were those that stated a wish or a new feature request, that is, a specific function that the app developer can add or incorporate in the new version of the apps. A total of 7,654 specific reviews were identified.

Next, the specific reviews were categorized to shortlist the innovation-related reviews. Consistent with previous studies, reviews concerning a new feature request were considered customer led innovation reviews [17][18].

A feature request refers to the wish or idea proposed by users on a new functionality that should be added but does not exist yet [6][25]. Examples of such review were "Needs to have a value for calories burned for strength training too" and "Missing a compatible version for Apple Watch". Keywords used to screen for bug reviews were "add", "please", "need", "prefer", "request", "suggest", and "wish" [17].

## Developer responsiveness

Developer responsiveness was measured by the time interval (number of days) from the first posted date of the user review to the update date when the bug was fixed or the advertisements were removed.

## Apps performance

Apps performance could be operationalized in a number of ways such as the number of downloads and apps ratings etc. In this study, apps performance was measured using the revenue generated from the apps during the research time frame. This allowed us to examine the financial impact on the apps developers more directly. Revenues could include purchases of apps, micro-transactions within an app or in-app advertisement (IADV) [8]. The revenues for each app was computed by a summation of the daily revenues for the research time frame.

## 3.3 Data Analysis

Regression Analysis was conducted to analyze the correlations in the measurement model. It enabled us to examine the significance and the magnitude of the impact from the independent variable (customer led innovation) on the dependent variable (performance in terms of revenues) [19].

As the data for customer led innovation took the form of written user reviews, steps were taken to convert the text data into numerical data. Each review was enumerated with a Python code respectively according to its match with the category of "feature request".

# 4 Findings

The results of the regression analysis were presented in table 1 below.

<table><caption>Table 1. Results</caption><thead><tr><th rowspan="2"></th><th colspan="2">Unstand. Coefficients</th><th>Stand. Coefficients</th><th rowspan="2">t</th><th rowspan="2">Sig</th></tr><tr><th>B</th><th>Std. Error</th><th>Beta</th></tr></thead><tbody><tr><td>Model 1 –<br>Customer Led Innovation</td><td>7,761.506</td><td>8,906.398</td><td>.154</td><td>.871</td><td>.388</td></tr><tr><td>Model 2 –<br>Customer Led Innovation</td><td>14,198.564</td><td>11,458.130</td><td>.281</td><td>-1.239</td><td>.021</td></tr><tr><td>Moderator of Customer Led Innovation</td><td>-37.904</td><td>14.565</td><td>-.590</td><td>2.602</td><td>.012</td></tr></tbody></table>

The overall model was significant with a p-value of 0.007. The overall R-square was 0.190, which was satisfactory and typical for exploratory research [19]. H1 was rejected (0.388), meaning that customer led innovation does not have a significant impact on apps performance in terms of revenues.
The moderating effect of developer responsiveness was supported (.012). It was negatively associated with the link between customer led innovation and apps revenues. With shortened response time, the impact of customer led innovation might increase the revenues further.

# 5 Discussion and Conclusion

User involvement has been well documented in the extant literature that the role of users plays an important part in enhancing the quality of software development. This presumption might not hold in the context of mobile apps, where hundreds or thousands of users may easily participate in the apps design through submitting reviews online. The volumes of user reviews might be hardly manageable and the usefulness of the reviews might also be impaired by the users' lack of technical expertise. It was questionable whether the users' creative solutions could lead to actual improvement of the apps performance. Our study therefore attempted to fill this gap in the literature and empirically examined the effect of customer led innovation over apps performance in terms of revenues.

Our findings provided empirical evidence on the value of user reviews on innovativeness of apps development. It was challenging for apps developers to imagine an exhaustive list of user wishes on new features. It was cost-effective to draw on user reviews to gain insights into novel features desirable by users. As customers acquired hands on experience with usage of the apps, they were more able to suggestion creative solutions. In other words, co-creation of apps with users should be encouraged to enhance innovativeness and hence the apps performance [9].

We also tested the moderating effect of developer responsiveness on the relationship between customer led innovation and apps performance. Though significant, the effect was not very strong. One plausible explanation is that developers may have to launch apps updates very frequently after addressing each new feature suggested by users. The recurring need to update the apps may be annoying to users [23] and discourage them from the continued usage of the apps. On the other hand, users might also grow impatient if new features took excessive time to be launched. Apps developers should therefore ensure user reviews on innovation were properly addressed within a reasonable time frame.

# References

1. Armerding, T. 2012. "Why Users Don't Often Upgrade Software When They Should". Available at https://www.csoonline.com/article/2132061/security-awareness/References 48 why-users-don-t-often-upgrade-software-when-they-should.html.

2. Barki, H., and Hartwick, J. 1989. "Rethinking The Concept of User Involvement," MIS Quarterly, pp.3-63.

3. Barlow, J., and Møller, C. 1996. A Complaint Is A Gift: Using Customer Feedback As A Strategic Tool. Berrett-Koehler Publishers.

4. Berger, C., Möslein, K., Piller, F., and Reichwald, R. 2005. "Co-Designing Modes of Cooperation at the Customer Interface: Learning From Exploratory Research," European Management Review (21), pp. 70-87.

5. Chen, N., Lin, J., Hoi, S., Xiao, X., and Zhang, B. 2014. "AR-miner: Mining Informative Reviews for Developers from Mobile App Marketplace," in Proceedings of the 36th International Conference on Software Engineering, pp. 767-778.

6. Cheung, K. W. 2013. "A Feature Request Is A Bug...Is a Dumb Detail," Available at https://www.getdonedone.com/a-feature-request-is-a-bug-is-a-task-is-a-to-do-is-a-dumb-detail/.

7. Comino, S., Manenti, F. M., and Mariuzzo, F. 2016. Updates Management in Mobile Applications. iTunes vs google play.

8. Ghose, A., and Han, S. P. 2014. "Estimating Demand for Mobile Applications in the New Economy," Management Science (606), pp.1470-1488.

9. Gustafsson, A., Kristensson, P., and Witell, L. 2012. "Customer Co-Creation in Service Innovation: A Matter of Communication?" Journal of Service Management (23), pp. 311-327.

10. Guzman, E., Ibrahim, M., and Glinz, M. 2017. "A little bird told me: Mining tweets for requirements and software evolution," In 2017 IEEE 25th International Requirements Engineering Conference, pp. 11-20.

11. Iacob, C., and Harrison, R. 2013. "Retrieving and analyzing mobile apps feature requests from online reviews," In 2013 10th IEEE Working Conference on Mining Software Repositories MSR, pp. 41-44.

12. Ives, B., and Olson, M. H. 1984. "User Involvement and MIS success: A Review of Research," Management Science (305), pp. 586-603.

13. Kaulio, M.A. 1998. "Customer, Consumer and User Involvement in Product Development: A Framework and A Review of Selected Methods," Total Quality Management (91), pp.141-149.

14. Khalid, M., Asif, M., and Shehzaib, U. 2015. "Towards Improving the Quality of Mobile App Reviews," International Journal of Information Technology and Computer Science (710), pp. 35.
15. Kujala, S. 2003. "User Involvement: A Review of the Benefits and Challenges," Behavior & Information Technology (221), pp. 1-16.

16. Kujala, S. 2008. "Effective User Involvement in Product Development by Improving the Analysis of User Needs," Behavior & Information Technology (276), pp. 457-473.

17. Maalej, W., and Hadeer, N. 2015. "Bug Report, Feature Request, or Simply Praise? On Automatically Classifying App Reviews," In 2015 IEEE 23rd International Requirements Engineering Conference.

18. Maalej, W., Kurtanović, Z., Nabil, H., and Stanik, C. 2016. "On The Automatic Classification Of App Reviews," Requirements Engineering (213), pp. 311-331.

19. Mooi, E., and Sarstedt, M. 2011. A Concise Guide to Market Research. Springer Berlin Heidelberg.

20. Panichella, S., Di Sorbo, A., Guzman, E., Visaggio, C. A., Canfora, G., and Gall, H. C. 2015. "How Can I Improve My App? Classifying User Reviews for Software Maintenance and Evolution," In 2015 IEEE International Conference on Software Maintenance and Evolution ICSME, pp. 281-290.

21. Prahalad, C. K., and Ramaswamy, V. 2013. The Future of Competition: Co-Creating Unique Value with Customers. Harvard Business Press.

22. Statista 2019. "Number of apps available in leading app stores as of 4th quarter 2019," Available at: https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/

23. Vanlea, K., and Rashidi, Y. 2016. "Tales of Software Updates: Software," In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3215-3226.

24. Vu, P. M., Nguyen, T. T., Pham, H. V., and Nguyen, T. T. 2015. "Mining User Opinions In Mobile App Reviews: A Keyword-Based Approach," In 2015 30th IEEE/ACM International Conference on Automated software engineering ASE, pp. 749-759.

25. Wiggins, N. 2015. "The difference between a Bug, Error and Feature," Available at: https://www.webigence.com/blog/the-difference-between-a-bug-error-and-feature.