Requirements knowledge acquisition from online user forums

IET SoftwareVolume 14, Issue 3 p. 242-253 ArticleOpen Access Requirements knowledge acquisition from online user forums Javed Ali Khan, School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this authorLin Liu, Corresponding Author linliu@mail.tsinghua.edu.cn School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this authorLijie Wen, School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this author Javed Ali Khan, School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this authorLin Liu, Corresponding Author linliu@mail.tsinghua.edu.cn School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this authorLijie Wen, School of Software Engineering, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this author First published: 01 June 2020 https://doi.org/10.1049/iet-sen.2019.0262Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Online discussion forums can be used for reflecting on the overall user experience of a system. If a user forum is well-structured, it can be a valuable source of requirements-related information, which can potentially be accommodated in the requirements engineering process to enhance the current and future software. However, presently, there are limited approaches for extracting such requirements-related information from the relevant community forums. To fill this gap, this study proposes an automated approach, which automatically identifies requirements information using natural language processing and machine learning. For this purpose, the authors analysed 3319 user comments collected from the seven discussion topics in the Reddit forum. Then, using a content analysis approach, they studied how frequently end-users submit such information across each discussion topic. Also, they developed an automated approach that identifies key stakeholders, who frequently contribute his rationales in the forum discussion. Further, they employed different machine learning algorithms to classify user comments into rationale elements of different types. The authors' results show that online forums, such as Reddit, can be a rich source of requirements elicitation. Also, machine learning is a promising tool to detect user's rationale and identify different kinds of requirements modelling elements. 1 Introduction For software products or services in a mass-market, understanding users' rationale is pivotal to keep the software evolve in accordance with the market needs. Also, engineers make key design decisions about software features by identifying the most important and relevant issues raised by the end-users. However, as users are often highly diversified, and could be distributed all over the world. Online user forums provide a place where knowledge and experience of individual users anywhere in the world could be shared quickly and easily. The development team could observe the arguments from different users, which will justify future design decisions, constitute rationale [1, 2]. In the traditional Software development life cycle, requirements are considered at the early beginning phase, where major decisions about product boundary, selection of features are made [3]. Today, in the iterative and agile project setting, decisions are often made on the run; hence, it is even more important to manage design and user rationale in order to keep track of reasons for certain design decisions [4]. The primary objective of managing user rationale is to capture reasons and provide justifications for a requirements related decision [5]. In addition, capturing user rationale improves requirements traceability, system documentation and system transparency [3]. In the literature, there is much emphasis on the importance of software requirements and design rationale management [6]. To decide what software features to provide for the market, product managers make trade-offs among multiple alternatives [7, 8]. Additionally, rationale management keeps track of the reasoning process for the system design [9] and facilitates software development teams in understanding past decisions being made [10]. Over the past decade, it is observed that researchers have shown a growing interest in analysing contents aggregated through the social media, app store, and user forums [11] due to their pervasive use by the crowd. The main focus for requirements engineering researchers is to identify useful and instinctive information related to requirements, e.g. features improvement requests [12, 13], bug reports [14], user experience with software applications [15], user comments in the user forums [16], non-functional requirements [17], and identifying user rationale elements [18, 19]. This intuitive information related to requirements can be found in mobile app stores, issue tracking systems, user forums, and Twitter [11]. To better understand rationale present in the user comments over the user forum, we conducted an exploratory study to the user forum on Reddit.com, to identify rationale elements and requirements related-information like, design alternatives, new features, issues, supporting or attacking claims [2]. In this paper, we further analyse the occurrence of each rationale element type, identifies the key stakeholders who are regular contributors of requirements-related information, and finally apply machine learning algorithms to check its potential in automatically identifying rationale elements at a different level of granularity. For this purpose, we use a content analysis technique [20], and manually annotated 3319 users comments, grouped by seven similar topics in the Reddit forum about 'google maps commute, directions, and location sharing' and then used the user annotated data to train and validate different machine learning algorithms to automatically identify requirements rationale elements. Our research provides quantitative proof that the online forum is a rich information source for requirements rationale. However, due to the large volume of user comments in the online forum, it is impossible to identify requirements-related information manually on a regular basis. In this paper, we use natural language processing and machine learning tools to classify user comments. Firstly, classifying user feedback into user comments with rationale or without rationale information; secondly, user comments with rationale are further classified into a new feature, issue, supporting and attacking arguments. There are four major contributions presented in this work: Firstly, we curated an experimental data set, which contains 3319 user comments from seven relatively relevant topics in the Reddit forum. Secondly, we conducted a detailed analysis of the user comments to identify rationale occurrence. Thirdly, we developed an automated process to identify key contributors for rationale and requirements-related information in the user forum. Finally, we compared different machine learning algorithms on their performance in automatically classify user comments. The main structure of the paper is as follows: Section 2 discusses the research questions. Section 3 introduces our proposed research method and the data set. Section 4 defines the target rationale elements. Section 5 introduces the statistics of the occurrence of rationale elements identified through content analysis techniques. Section 6 introduces the approach to identify key stakeholders, who frequently contribute their rationale. Section 7 elaborates on our machine learning experiments on classifying user comments. Section 8 reports the research findings and threats to validity. Section 9 discusses related work, and Section 10 concludes the paper and discusses future work. 2 Research questions In this section, we will elaborate on the main research questions to be answered through our research study. Research Question 1: What is the occurrence pattern of the rationale elements in the Reddit forum? The goal of this research question is to examine the Reddit forum and identify how often the rationale elements appear in the user comments, which in turn answers whether user forums can be considered as a good source of rationale information. Besides, identify requirements-related information suggested by end-users, e.g. new features or alternatives. To answer this question, we collected user comments from the Reddit forum and then annotated user comments manually to identify the occurrence of rationale element. Research Question 2: How to automatically identify key stakeholders who frequently contribute rationale and requirements-related information in the user forum? In this research question, we will automatically identify key stakeholders who frequently contribute rationale and requirements-related key information. For this purpose, we developed an automated algorithm to identify key stakeholders. Research Question 3: How well machine learning algorithms perform in automatically classifying user comments into rationale elements and requirements-related information. The classification task is performed at binary and multiple levels. Particularly, we are keen to answer the following research questions. Can machine learning algorithms efficiently and accurately classify user comments into the ones with rationale ones without rationale?. Can supervised machine learning algorithms efficiently and accurately further classify user comments into different types of rationale elements?. 3 Research method overview Our research methodology is comprised of two main steps: data collection to support our proposed approach and data analysis to answer the above-mentioned research questions, as shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Our research methodology 3.1 Research data set collection To run our approach, we collected user comments from the Reddit [http://www.reddit.com] forum using python PRAW [https://praw.readthedocs.io/en/latest] API. Our data set reflects the structure of the user conversation in the user forum. It contains user statements, author names, associated user votes, the commenting date and time, comment id, and parent id. In the Reddit forum, usually, an end-user starts a discussion thread, either by sharing his experience with a vendor application, or requests for a new feature, or report an issued. For this discussion thread, other users post comments supporting or opposing the opinion of each other. Also, they can record a new issue encountered by them, or suggests a new feature or an alternative. In this paper, we collected user comments for seven relevant topics in the Reddit forum, as shown in Table 1. The actual number of the user comments displayed in the Reddit forum with each topic is different from our data set, e.g. the actual user comments gathered for the following topic, 'google maps introduce location sharing' is 701, while the user comments shown in Reddit forum is 781. This is because there were some user comments deleted by the author or by the forum administrator. Hence, we have to delete their corresponding child comments to reflect the original structure of user conversation in the forum. In total, we have collected 3319 user comments from the seven discussion topics of the user forum. The details of each user discussion topic are shown in Table 1. Table 1. Summary of the seven sample data sets ID. Discussion topic Number of statements Topic links 1 Google maps introduces location sharing 701/781 https://www.reddit.com/r/Android/comments/60v5za/ 2 take control of your commute with Google Maps 401/435 https://www.reddit.com/r/Android/comments/9kfdyl/ 3 all together now: group planning on google Maps 268/293 https://www.reddit.com/r/Android/comments/9j1v84/ 4 I wish Google Maps had an option to give me less verbose directions, but not entirely mute. 467/552 https://www.reddit.com/r/Android/comments/4bxtcm/ 5 Google Maps will soon allow you to share your ETA, add a shortcut to routine destinations, and create a map of your location history 583/803 https://www.reddit.com/r/Android/comments/4lpvk6/ 6 Google Maps now warns you if you are navigating to a place that will be closed when you arrive 511/560 https://www.reddit.com/r/Android/comments/39yslr/ 7 Google Maps for android is finally rolling out multi-waypoint directions 388/445 https://www.reddit.com/r/Android/comments/4qg62/ An excerpt from the end-users conversation in the original discussion topic 'Google Maps Will Soon Allow You to Share Your ETA, Add a Shortcut to Routine Destinations, and Create a Map of Your Location History' is shown in Fig. 2. In the example, end-users are discussing the new features introduced in the google map mobile application. The end-users discussion snippet in Fig. 2 shows that a user 'Rocketpinion' asked for a new feature in response to the main discussion topic. The new feature gets 250 points, which is an accumulative sum of supporting votes and opposing votes, gained from the other registered users in the Reddit forum. For which, 'Minimalfish' registered a supporting comment while 'Am3n' gives a neutral comment. The two claims received 21 and 1 points, respectively. The end-user claims identified as 'neutral' are considered as irrelevant information to the topic under discussion, hence ignored. During the automated classification of user comments, the irrelevant user comments will be filtered out as user comments without rationale, as explained in Section 7. Another user 'Jardolam_' posted an issue against the main discussion topic, which asked a clarification question. The issue gets 1 point in response from another user in the Reddit forum. Finally, 'SkyrO_' attacks the main discussion topic, which gets 0 point. Fig. 2Open in figure viewerPowerPoint User discussion excerpt in the Reddit forum 3.2 Data analysis To answer Research Question 1, we applied content analysis techniques [20]. The user comments annotation process is explained in Section 5. Secondly, we developed an automated approach to identify the key stakeholder in the user forum, who contributes rationale and requirements-related information more frequently than the other end-users to answer Research Question 2. For this purpose, we extracted the author's name from the same data set collected from the Reddit user forum. The process is elaborated in Section 6. Finally, to answer Research Question 3, we experimented with different machine learning algorithms to compare their performance, firstly, by classifying users comments into the ones with rationale and the ones without rationale, secondly, classifying those user comments which are identified with rationale into fine-grained rationale elements, such as new features or alternatives, issues, supporting and attacking arguments or claims. The machine learning process is explained in Section 7. 4 Rationale elements In our previous work [2], we conducted a detailed analysis of the user comments in the Reddit forum and identified different rationale and requirements-related information. The identified rationale element types are a new feature or a design alternative, an issue, supporting and attacking arguments. The identified new features are of great importance for the requirements engineers, which they considered during the next release to the market. The detailed explanation of the rationale elements along with their example, is shown in Table 2. For this study, we are considering the same rationale elements types to run our proposed research approach. Each rationale element type is elaborated below. Table 2. Types of rationale elements ID. Example user statements from Reddit Label Definitions of the label 1 'I want google maps to have an option to avoid main roads, to take the side road instead. That's really about it.' feature end user suggests or demand for a new features, or suggests an alternative solution 2 'This is what I loved about Waze. I could share my progress with someone. Google Maps…….Finally!', 'This is so cool' claim-supporting end-users submits a claim in support of an issue, or a feature 3 'Just wait. I'm pretty sure Google will remove Location sharing with Maps 10 or 11', 'Say goodbye to your battery' claim-attacking end-users submits a claim to attack an issue, or a feature proposed 4 'Do they ever add anything to Auto? That thing had stagnated for months.', 'Are there really uses for that?' issue problem that needs further discussion and elaboration 4.1 New feature or design alternative During an ongoing discussion in the Reddit user forum, at any point, end-users can suggest a new feature or alternative in response to the main discussion topic, or raise an issue, or post a relevant statement. Such information is of great importance for the requirements engineers, which they will consider for the next release of software under discussion [2, 14]. As can be seen in Fig. 2, where a user named 'Rocketpinion' suggests a new feature 'How about letting me turn on 'avoid tolls' by default, instead of having to turn it on every time i request a route in response' to the main discussion topic 'google maps will soon allow you to share your ETA, add a shortcut to routine destinations, and create a map of your location history'. 4.2 Claim-supporting In the user forums, users provide supporting statements for the alternative features, issues, or the main discussion topic. Users express their support with such statements. For example, in Fig. 2, an end-user provides a supporting argument 'Exactly, why is this not at the top? it seems so easy to implement' in response to an alternative feature 'How about letting me turn on 'avoid tolls' by default, instead of having to turn it on every time i request a route' proposed by the crowd-user in the Reddit user forum. 4.3 Claim-attacking Similarly, users provide attacking or opposing arguments for alternative features, issues, or the main discussion topic. Crowd-users express their disagreements to the main discussion topic, a new feature, or an issue with such statements. For example, in Fig. 2, an end-user provides attacking argument 'Google jacking off because of all the delicious new data they will get, lol' in response to the main discussion topic 'google maps will soon allow you to share your ETA, add a shortcut to routine destinations, and create a map of your location history' in the Reddit user forum. 4.4 Issues During an ongoing discussion in the Reddit forum, end-users post certain challenges or questions in response to the main discussion topic, a new feature, or an issue. These questions or challenges need further discussions and elaborations, which in turn helps requirements engineers to understand the difficulty faced by the users in using the software application under discussion [2]. For example, as shown in Fig. 2, where a crowd-user asks a question 'What is an ETA?' in response to the main discussion topic 'google maps will soon allow you to share your ETA, add a shortcut to routine destinations, and create a map of your location history' in the Reddit user forum. 5 Frequency of rationale elements To identify the frequency of rationale elements in user comments, we performed the following steps. 5.1 Experiment data set preparation To find rationale elements in user forums, we manually annotated the user comments in our data set using a content analysis technique [20]. For this purpose, the first two authors of the paper individually inspected the user comments, collected from the Reddit user forum in our data set. The annotation steps are performed in the following steps. 5.1.1 Developing the annotation guide To minimise the disagreement between the coders, we have developed a coding [https://drive.google.com/open?id=1nEYiu2ZP72n68fOYvlJMV4LF64_pk1fG] guideline, which contains rationale elements definitions along with their detailed examples. The sole purpose of this step is to make sure that the two coders share a common understanding of the rationale concepts. The annotation guide for the coders was developed in several iterations. 5.1.2 Coding of end-users comments To manually annotate the user comments in our data set, the coding process was carried out using the Microsoft comma-separated values (CSV) program, the screenshot of the coding file is shown in Fig. 3. Each coder received the annotation guide and complete the CSV document containing user comments from the user forum. It can be seen in Fig. 3 that, the columns 'Comment_Text', 'Comment_author', 'Comment_Id', and 'Parent_Id' reflects users information in our data set while the column 'have_rationale' indicates that whether the user comments contain rationale or requirements-related information, the coders can select 'yes' if it contains rationale or requirements-related information and vice-versa. Finally, the column 'Rationale_Type' indicates the type of elements which is to be selected by the coders for the user comments if their corresponding value is marked as 'yes' in the 'have_rationale' column. The types of elements are feature, issue, claim-supporting, and claim-attacking. Fig. 3Open in figure viewerPowerPoint Sample of user coding The two coders annotated each user comment in our data set independently. The average time taken by the two coders to annotate the user comments in our data set was reported as 15 h, which reflects the great exertion required to manually identify rationale elements from the user comments. 5.1.3 Disagreement resolution Disagreement occurs between the two coders when they labelled the same user comment as different types or one coder identifies a user comment as irrelevant. The disagreement between the two coders is resolved by discussion. After annotating the user comments, we merged the completed coding results from the two annotators to calculate the inter-coder agreement between them. The average inter-coder agreement between the two coders was 86% while Cohen's kappa for the two coders was 65%, which is considered as a substantial agreement on Cohen's kappa scale. 5.1.4 Annotation results Fig. 4 shows the distribution of rationale occurrence in user comments across each discussion topic, identified by the coders through the annotation process. It can be concluded from Fig. 4, the user comments in the user forum are considered as an important source of rationale. On average, 74% of user comments contain rationale and requirements-related information out of the 3319 user comments, collected from seven topics in the Reddit forum, as shown in Table 1 and vice-versa. Such a large amount of rationale information's are of great importance for the software development teams [7]. The details of each topic number in Fig. 4 are mentioned earlier in Table 1. Fig. 4Open in figure viewerPowerPoint Frequency of user comments Table 3 further elaborates on the frequency of rationale elements and requirements-related information identified by the two coders from the user comments against each discussion topic. To summarise, we found in our data set, that the most prominent rationale element present is 'claim-supporting', which is identified as 33% of the overall user comments (2465). The end-users submit the supporting arguments in response to the main discussion topic, a new feature proposed, or an issue reported by other end-users in the Reddit forum. The second most prominent rationale element identified in our data set is the 'new feature or alternatives' proposed by the end-users. The new feature or alternative rationale element identified is 29%. Such types of information's are of great importance for requirements engineers, which they considered for the future release of the software [2, 21]. It proves that the user comments in the Reddit forum can be considered as a rich source of information for requirements engineers. 24% of the user comments were identified as 'attacking arguments' in our data set, which are submitted by the end-users to oppose the main discussion topic, a feature, or an issue reported. Finally, the least rationale element identified in our data set is an 'issue', which contains 14% of overall user comments identified with rationale. Such information is useful for requirements engineers, which help them in understanding the difficulty faced by the end-user while using the software application under discussion [2]. Besides, Table 3 shows the distribution of rationale and requirements-related information against individual discussion topics taken from the Reddit forum. Table 3. Rationale elements frequency against each topic Rationale element. Topic 1, % Topic 2, % Topic 3, % Topic 4, % Topic 5, % Topic 6, % Topic 7, % Total, % features 24 24 28 42 35 24 26 29 issue 13 18 16 9 16 11 15 14 claim-supporting 39 32 25 32 26 34 37 33 claim-attacking 24 26 31 17 23 31 22 24 6 Identifying key stakeholders We developed an automated approach to identify key stakeholders, who are frequently contributing rationale and requirements-related information in the Reddit forum. For this purpose, we used collections [https://docs.python.org/2/library/collections.html.] and itertools [https://docs.python.org/2/library/itertools.html.] libraries of python environment to identify the key stakeholders, their rationale and requirements-related information contribution. In total, 1715 unique users contributed 2465 user comments across seven relatively similar topics in the Reddit forum, which are identified with rationale information. Amongst these 1715 crowd-users, there were 116 end-users whose names were missing in our data set, as they have not submitted their names while submitting the feedback in the Reddit forum. Therefore, we replace their empty author names with 'anonymous' in our data set. The number of feedbacks submitted by the end-users in our data set is shown in Fig. 5. In which, 1341 end-users contributed feedback only once, while 245 end-users contributed twice in our data set, resulting in 490 user comments overall. Also, there were 68 end-users, whose contribution was thrice; similarly, 35 end-users submitted four times feedback in the Reddit forum. Finally, there were some key end-users, which actively participate in the discussion forum and contributed rationale and requirements-related information frequently, such as 9 end-users contributed feedback 5 times, 5 end-users have contributed 6 times, 4 end-users have contributed 7 times, 2 end-users have contributed 8 and 9 times respectively, while 1 end-user has contributed 10, 13, and 14 times for each topic. Fig. 5Open in figure viewerPowerPoint No. of times feedback submitted by end-users As shown in Fig. 6, our automated approach was able to identify the details of each end-user rationale elements submitted against the discussion topic in the Reddit forum. It helps identifying key stakeholders, who frequently contribute rationale and requirements-related information. Fig. 6 shows the details of an end-user named 'Shiquoxian', whose contributed 13 rationale and requirements-related information in the Reddit forum. Additionally, he is considered to be a key stakeholder as he contributed the second most feedback in our data set. He suggested three new features in response to the main discussion topic, reported an issue encountered by using the software application under discussion, given three supporting and six attacking arguments either in response to the main discussion topic, a new feature proposed by other users, or an issue encountered by other users. Fig. 6Open in figure viewerPowerPoint Distribution of rationale elements of an end-user (Shiguoxian) To help requirements engineers, our automated approach identifies key stakeholders who frequently contribute requirements-related information in terms of new features in the Reddit forum. Identifying key stakeholders is pivotal in requirements engineering for the success of market-based software applications [22]. In contrast, key stakeholders become frustrated if their contribution is ignored [23], influencing customer satisfaction of software products [24]. Fig. 7 shows the details of key end-users who frequently suggest new features in the Reddit forum. Among these key users, 'dlerium' becomes the most influential person as, he suggested four new features related to the main discussion topic, reported three issues encountered with the google map application and gave one supporting argument in favour of the main discussion topic, a new feature or an issue encountered. The other four key end-users (huffalump1, wayoverpaid, johnrclem, and ProfWhite) have suggested three new features as well. Such requirements related information is very useful for requirements engineers, which can be utilised for a future release of software application [2]. Fig. 7Open in figure viewerPowerPoint Key stakeholders contributing features Similarly, our proposed approach helps requirements engineers to give more focus and attention to those crowd-users which frequently reports issues related to the software under discussion in the Reddit for