What have we learnt from the challenges of (semi‐) automated requirements traceability? A discussion on blockchain applicability

IET SoftwareVolume 15, Issue 6 p. 391-411 REVIEWOpen Access What have we learnt from the challenges of (semi-) automated requirements traceability? A discussion on blockchain applicability Selina Demi, Selina Demi Faculty of Computer Sciences, Østfold University College, Halden, NorwaySearch for more papers by this authorMary Sanchez-Gordon, Mary Sanchez-Gordon orcid.org/0000-0002-5102-1122 Faculty of Computer Sciences, Østfold University College, Halden, NorwaySearch for more papers by this authorRicardo Colomo-Palacios, Corresponding Author Ricardo Colomo-Palacios [email protected] orcid.org/0000-0002-1555-9726 Faculty of Computer Sciences, Østfold University College, Halden, Norway Correspondence Ricardo Colomo-Palacios, Faculty of Computer Sciences, Østfold University College, Halden, 1757 Norway. Email: [email protected]Search for more papers by this author Selina Demi, Selina Demi Faculty of Computer Sciences, Østfold University College, Halden, NorwaySearch for more papers by this authorMary Sanchez-Gordon, Mary Sanchez-Gordon orcid.org/0000-0002-5102-1122 Faculty of Computer Sciences, Østfold University College, Halden, NorwaySearch for more papers by this authorRicardo Colomo-Palacios, Corresponding Author Ricardo Colomo-Palacios [email protected] orcid.org/0000-0002-1555-9726 Faculty of Computer Sciences, Østfold University College, Halden, Norway Correspondence Ricardo Colomo-Palacios, Faculty of Computer Sciences, Østfold University College, Halden, 1757 Norway. Email: [email protected]Search for more papers by this author First published: 04 July 2021 https://doi.org/10.1049/sfw2.12035Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Over the last 3 decades, researchers have attempted to shed light into the requirements traceability problem by introducing tracing tools, techniques, and methods with the vision of achieving ubiquitous traceability. Despite the technological advances, requirements traceability remains problematic for researchers and practitioners. This study aims to identify and investigate the main challenges in implementing (semi-)automated requirements traceability, as reported in the recent literature. A systematic literature review was carried out based on the guidelines for systematic literature reviews in software engineering, proposed by Kitchenham. We retrieved 4530 studies by searching five major bibliographic databases and selected 70 primary studies. These studies were analysed and classified according to the challenges they present and/or address. Twenty-one challenges were identified and were classified into five categories. Findings reveal that the most frequent challenges are technological challenges, in particular, low accuracy of traceability recovery methods. Findings also suggest that future research efforts should be devoted to the human facet of tracing, to explore traceability practices in organisational settings, and to develop traceability approaches that support agile and DevOps practices. Finally, it is recommended that researchers leverage blockchain technology as a suitable technical solution to ensure the trustworthiness of traceability information in interorganisational software projects. 1 INTRODUCTION The concept of requirements traceability (RT) was introduced more than 30 years ago by researchers and practitioners [1]. Traceability has evolved from just tracing requirements to implementation and test artefacts [2], to playing a significant role in various software and systems' engineering activities, such as change and defect management [2, 3], project management [3], validation and verification [4], software maintenance [5], and impact analysis [6]. Traceability is particularly important in safety-critical systems [7-9] as it ensures safety, which is crucial for systems whose failure may result in the loss of life, loss or misuse of sensitive information, and major financial loss. According to Gotel and Finkelstein [10], RT refers to 'the ability to describe and follow the life of a requirement, in both a forwards and backwards direction (i.e., from its origins, through its development and specification, to its subsequent deployment and use, and through all periods of on-going refinement and iteration in any of these phases)'. The typical traceability process model is described by Gotel et al. [11] and entails the creation, maintenance and use of trace links within the scope of a defined traceability strategy. The manual creation of trace links poses the risk of inconsistencies, particularly in complex software projects with a variety of artefacts and relations among them. To reduce the burden of manual tracing tasks, which are time consuming and tedious, automation is of major importance [12]. In fact, enhancing the automatic degree of tracing activities was identified as the second most important research topic in a recent systematic literature review (SLR) on RT technologies [13]. In this regard, information retrieval techniques have been widely shown to support the (semi-)automated creation of trace links [14]. However, these textual-based techniques tend to generate traceability matrices with high recall and low precision [15]. Due to the high amount of false positives generated by these techniques, human analysts are required to vet candidate trace links and make the final decision on whether to accept or reject these trace links [16]. Despite the vast body of knowledge that exists on traceability, traceability practices are far from being mature [17]. Over the years, RT in general and (semi-)automated RT in particular have been widely identified as problematic by the industry [18-23]. According to the NaPIRE (Naming the Pain in Requirements Engineering) survey, 32.4% (158 out of 488) of the respondents positioned 'missing traceability' in the top 15 requirements engineering (RE) challenges [24]. This is a concerning value, given the consequences that can reverberate throughout the software development process and product due to the lack of traceability, such as less maintainable software or defects due to inconsistencies [17]. Traceability researchers strived to explore requirements traceability challenges, which led to a lack of implementation or an implementation of RT in a haphazard manner [21, 25-27]. The foundational work in this dimension is laid by Gotel and Finkelstein [10]. These authors attribute the poor requirements traceability practices to the lack of a common definition and the perception that traceability is expected to address conflicting problems in the context of different users, projects and tasks. Further, the Centre of Excellence for Software Traceability (CoEST) published a technical report in 2011 [28], in which traceability researchers and practitioners within the CoEST presented their vision for traceability and identified eight challenges that need to be addressed to achieve the vision. The last challenge entails making traceability ubiquitous, that is, traceability that is built into the system or software engineering process. This challenge has been named the grand challenge of traceability, as it requires progress with the seven remaining challenges: purposed, cost-effective, configurable, trusted, scalable, portable, and valued [28]. Our study aims to contribute to this body of knowledge by identifying and classifying challenges of (semi-)automated RT, as reported in the recent literature. To achieve this aim, we carried out a systematic literature review and reviewed 70 primary studies. To the best of our knowledge, the last similar literature review was conducted in 2009 by Winkler and Pilgrim [17]. Therefore, there is a need for an updated and comprehensive study that intends to provide a holistic view of (semi-)automated RT challenges (for related works, see Section 2). The contributions of this study are as follows: (i) the identification of 21 domain-agnostic challenges and their classification into five categories, (ii) the proposal of blockchain as a suitable solution to ensure reliability and availability of traceability across organisational boundaries, (iii) the identification of research gaps and the call for further research efforts in the following dimensions: distributed traceability, human factors, traceability approaches in agile and continuous software engineering, and more exploratory studies in order to enhance the comprehension of traceability practices. The remainder of the study is organised as follows: We present the related works in Section 2 and the research methodology in Section 3. In Section 4, we introduce the results in terms of the primary studies' overview and the identification of the challenges. These results are discussed in relation to the related work in Section 5, along with directions for future research and validity threats. Finally, we conclude the study in Section 6. 2 RELATED WORK A summary of the related works is provided in Table 1 and it indicates that the last comprehensive literature review focussed on RT challenges was conducted in 2009 and published in 2010. In what follows, we present the related works: TABLE 1. Related works Study reference Author(s) Publication year Focus Research method # Studies/Organisations Period of SLR [29] Ramesh 1998 Influencing factors on the use and adoption of RT Evaluation research 26 organisations - [18] Blaauboer et al. 2007 Influencing factors on the decision to use RT Evaluation research 1 organisation - [30] Kannenberg and Saiedian 2009 Challenges of RT Review - - [17] Winkler and Pilgrim 2010 Traceability in RE and model-driven development and their challenges Review - - [26] Torkar et al. 2012 Factors hindering the implementation of RT SLR and evaluation research 52 primary studies + 2 organisations 1997–2007 [2] Regan et al. 2012 Barriers faced when implementing RT Review 8 primary studies 2005–2011 [31] Nair et al. 2013 The evolution of the RT research in the RE conference SLR 70 primary studies 1993–2012 [4] Mustafa and Labiche 2017 Modelling traceability among heterogeneous artefacts SLR 330 primary studies 2000–2016 [13] Wang et al. 2018 RT technologies SLR 114 primary studies 2000–2016 [27] Maro et al. 2018 Traceability challenges on the automotive domain Tertiary study + multi-vocal literature review + evaluation research 24 secondary studies + 245 sources + 1 organisation 2007–2017 Abbreviations: RE, requirements engineering; RT, requirements technology; SLR, systematic literature review. Ramesh [29] conducted surveys with participants in 26 organisations and classified them into low-end and high-end users of traceability, according to their underlying motivation to implement traceability. While low-end users perceive traceability as a mandate, high-end users perceive traceability as an important quality attribute of system and software engineering. This study also presents factors influencing the practice of requirements traceability and categorises them into environmental, organisational and system development contexts. However, the emergence of distributed development and agile paradigms introduces the need for a new study in this dimension. Blaauboer et al. [18] conducted a case study in order to identify factors influencing the decision to use requirements traceability. Given the management perspective of this study, the following factors were identified: organisation awareness, customer awareness, return on investment, stakeholder preferences, and process flow. The authors emphasised the lack of awareness among software project leaders regarding traceability. Conversely, the scope of our study is not limited to the management perspective. Kannenberg and Saiedian [30] identified the following challenges of implementing RT: cost, change management, different stakeholders' viewpoints, organisational problems, and poor tool support. They observed that many organisations struggle to comprehend the importance and benefits of traceability and suggested organisational changes and better tool support in order to reap these benefits. Winkler and Pilgrim [17] carried out an extensive literature review that was conducted in 2009 and published in 2010. The goal of this study was to explore traceability in requirements engineering and model-driven development. These authors pointed out natural, technical, economical, and social factors that hinder the implementation of traceability practices in the industry. Additionally, they recommended further research on improving the human factor in traceability, enabling distributed traceability and providing support for practical problems in the industry. Our study follows a similar approach but provides an updated view of (semi-)automated RT challenges, as reported by the recent primary studies. Torkar et al. [26] identified requirements traceability tools, techniques and challenges by reviewing studies published within the period 1997–2007. According to these authors, cost is the main factor that hinders the implementation of adequate traceability practices. Regan et al. [2] reported 11 traceability challenges that were categorised into management, social, and technical issues. Some of the reported challenges are cost, lack of guidance, political issues, and tool issues. Although these authors advocate the relevance of the challenges in both general and safety-critical domains, they acknowledge that the need to ensure accountability in safety-critical domains poses further complexities to the implementation of automated traceability. Nair et al. [31] explored the evolution of requirements traceability research by reviewing studies published in the requirements engineering conference (RE) within the period 1993–2012. This review addressed various aspects of requirements traceability, including their challenges. The authors report an increasing interest in automated traceability and suggest traceability visualisation as an area for further research. On the contrary, our systematic literature review (SLR) covers recently published studies, which are not constrained to a specific conference or journal. Mustafa and Labiche [4] reviewed studies that focussed on traceability in heterogeneous systems. The authors report a minimal research effort on modelling traceability among heterogeneous artefacts and call for more research in this dimension. Our study takes into account requirements traceability among heterogeneous artefacts, albeit in a more general approach. Furthermore, Wang et al. [13] carried out a systematic literature review to identify RT technologies and their respective challenges. Their findings indicated the following challenges: automated, trustworthy, lightweight, scalable, dynamic, tracing non-functional requirements, value-perceptible, cost-effective, coordinated, and expressible. Our study, however, takes a different approach by explicitly identifying challenges throughout the tracing process, independent of specific technologies. For instance, our study takes into account the human factor in traceability, which is outside the scope of Wang et al.'s review [13]. Recently, Maro et al. [27] conducted a tertiary literature review, a multi-vocal literature review and a case study in the automotive domain. They identified 22 traceability challenges and categorised them into 7 groups: human factors, uses of traceability, knowledge of traceability, tool support, organisation and processes, measurement of traceability and exchange within and across organisations. Instead of that, we carry out a systematic literature review that intends to identify and classify domain-independent challenges of (semi-)automated RT. Table 1 summarises the aforementioned related works and points out the main differences in terms of the publication year, research focus, research method, number of studies/organisations and the period of the SLR. We notice a limited number of studies that focus on investigating the challenges of RT practices, which are referred to as evaluation research according to the requirements engineering paper classification proposed by Wieringa et al. [32]. Additionally, the number of organisations considered by these four studies is low, except from Ramesh's [29] study. The other set of studies are literature reviews (tertiary, systematic literature reviews or multi-vocal reviews) that explore RT challenges as reported by the previous literature. The number of studies included in these reviews range from eight [2] to 330 primary studies [4]. Although there are few literature reviews related to this topic, the last one that aims to explore domain-agnostic challenges of implementing (semi-)automated requirements traceability was carried out by Winkler and Pilgrim [17] in 2009. Therefore, as mentioned before, there is a need for an updated comprehensive study in this area. 3 RESEARCH METHOD We carried out a systematic literature review that relies on the guidelines for performing systematic reviews in software engineering proposed by Kitchenham [33]. In what follows, the review process is thoroughly explained [33]. Planning the review The review protocol consists of relevant research question(s), search strategy and inclusion/exclusion criteria. We developed the protocol via brainstorming sessions and performed searches separately in two rounds. The feedback from these rounds was used to uncover problems in the initial version of the protocol and to improve its effectiveness. 3.1.1 Research question This study is aimed at identifying and classifying the challenges in implementing (semi-)automated RT, as reported by the recent literature. Therefore, we raise the following research question: RQ: What are the challenges in implementing (semi-)automated RT that have been reported in literature? The research question was designed according to the question structure proposed by Kitchenham [33]. In this regard, we define requirements traceability as the population or the subject of the study, (semi-)automation as the intervention and challenges as the outcomes. 3.1.2 Search strategy After setting the scope of this study, we defined the search strategy, search string and online databases, based on our experience with SLRs in software engineering [34-36]. This process was done prudently to ensure the precision of search strings and, consequently, minimise threats, such as inappropriate results, missing relevant studies or an increased overhead [37]. Regarding search strategies, Jalali and Wohlin [38] compared two main search strategies for systematic literature reviews—database search and backward snowballing—and concluded that there are no significant differences between the conclusions and patterns derived from these approaches. In this regard, we employed database search as the first-step search strategy, since it is the recommended approach in software engineering [38]. Nonetheless, Jalali and Wohlin [38] also pointed out that database searches lead to a lot of noise, meaning a higher number of irrelevant studies than included studies. Therefore, to mitigate the potential risk of overlooking relevant studies, we also applied the backward snowballing technique, complementary to the database search [33]. The reference lists of the studies selected were scanned and went through a three-stage selection process (see Figure 1). FIGURE 1Open in figure viewerPowerPoint Three-stage selection process The formulated query aims to identify the wide spectrum of available literature focussed on (semi-)automated requirements traceability. We identified three main terms and their respective forms. The first two terms are related to the subject of the study, which is requirements traceability. We also used the term 'software', since we noticed that researchers tend to overlook the term 'requirements traceability' and use 'software traceability' instead. Moreover, we incorporated the term 'tracing', as an alternative form of 'traceability'. The last group of terms consists of automated and (semi-)automated, which have been used interchangeably in the RT literature. It is noteworthy that we conducted trial searches with other terms, for example, 'semi-automatic', 'semi-automation', and 'assisted'; however, we did not identify further relevant studies. We concatenated these three groups of terms using the Boolean operator 'AND', and their forms using the Boolean operator 'OR'. The final search string was (automated OR semi-automated) AND (requirements OR software) AND (traceability OR tracing). Given that different search engines have different requirements, for instance, some search engines do not allow nesting, we tailored our search string to these requirements. The search string was issued in five major online databases, as suggested by Kuhrmann et al. [37]. The search string was executed in each of the databases in two rounds: first, in February 2020 and finally in June 2020. We delimited the time period to 2009–2019 because the last literature review similar to our approach was conducted in 2009 (see Section 2). We also employed the filter 'Computer Science', when available. Finally, a set of 4530 studies was retrieved, as shown Table 2. TABLE 2. Overview of studies Database Initial search First exclusion Final exclusion IEEE Xplore 267 52 36 ACM Digital Library 246 13 8 Springer Link 1470 17 8 Science Direct 1123 7 5 Wiley Online 1424 3 3 Selected studies 4530 92 60 Backward snowballing 10 Primary studies 70 3.1.3 Inclusion and exclusion criteria Given the high amount of studies retrieved from the database search, we followed a rigorous and reproducible selection process by defining the inclusion and exclusion criteria [37] as follows: A) Inclusion criteria The study must be published within the period 2009–2019. The study must be available as a full-text article. The study is included if it identifies or addresses at least one challenge of (semi-)automated requirements traceability. B) Exclusion criteria The study is excluded if it is not written in English. The study that focusses on traceability in domains other than software engineering is excluded. The study is excluded if it focusses on manual traceability. If the same article is available in more than one database, the versions available in databases other than the one that provides the article for download are excluded [37]. If a conference article is followed by a journal article, the conference article is excluded (given that a journal article is a higher-value publication and extends the conference publication) [37]. Conducting the review 3.2.1 Selection of studies The selection process consisted of three stages, as depicted in Figure 1. First, we retrieved studies from online databases using the search strategy defined in Section 3.1.2. This first stage generated a total of 4530 studies. After removing duplicates, we assessed the title, abstract, and introduction/conclusion (when necessary) of the studies, against the inclusion/exclusion criteria. In this stage, we selected 92 studies and collected their full texts using the reference manager, Zotero. Further, we independently read the full texts and made individual decisions regarding the final selected studies, based on their quality criteria. Despite some differences in our evaluations, concordance between quality scores was achieved. Disagreements were resolved through discussions. For instance, studies such as [26, 27], which use both the literature review and empirical research methods, were included after discussions because they were evaluated to provide relevant findings. After achieving consensus, we selected 60 final primary studies and excluded the remaining 32 studies (for the list of excluded studies, see [39]). Studies were excluded if they focused on benefits of RT, for example, [5, 25]; if they were short versions of other studies (in such a case we selected the extended version), for example, [40, 41]; if they were secondary studies, for example, [4, 13, 30]. Further, we scanned the reference lists of the selected primary studies and retrieved 20 studies. These studies went through the second stage of the selection process and 10 of them were selected. Finally, a total of 70 studies composed the final set of primary studies (see Appendix). 3.2.2 Study quality assessment According to Kitchenham [33], an agreed-upon definition of 'quality of studies' does not exist. Bearing this in mind, we formulated quality questions based on our experience with SLRs in software engineering [34-36] and assessed the selected studies accordingly (see Table 3). We evaluated the studies with the following scores: 0 (does not fulfil the criteria), 0.5 (partially fulfils the criteria), and 1 (fulfils the criteria). A threshold of at least 60% of the maximum score (>3 out of 5) was chosen, to ensure the quality and relevance of the selected studies. Each of the authors computed the quality score of the studies independently. In order to ensure the consistency of the study quality assessment, we computed Krippendorff's alpha (α). The value of this parameter was 78%, which dictates a similarity in the interpretation of data among the co-authors. However, few discrepancies were identified, discussed and resolved through consensus. TABLE 3. Quality questions ID Quality question 1 Does the study clearly define the aim? 2 Is the chosen research method appropriate to the research questions? 3 Is the research methodology explained in detail ensuring reliability, internal/external validity, and replicability? 4 Does the study discuss/identify/address any challenges of (semi-)automated requirements traceability by using empirically measured data? 5 Do the conclusions answer the research question(s)? Data extraction and synthesis We used the reference manager Zotero to automatically extract data for each study regarding the following attributes: the title of the study, author(s) name(s), date of publication, and the source of publication (journal, conference, symposium, and workshop). The automatic extraction of this data prevents inconsistencies, which may lead to erroneous analysis and interpretation of the findings. Two of the authors of the study extracted the data independently, by means of an extraction form. The extraction form and the data are available online as archived open data [39]. Other attributes of the data extraction form were inserted manually. For instance, research methods were classified into experiments, surveys and case studies, using the definitions of empirical research methods in software engineering [42]. According to Wohlin et al. [42], these research methods are non-competing; on the contrary, they can be used together in order to enable more informed decisions in software engineering. Therefore, we also investigated combinations of these research methods. In addition, the data extraction form contains the quality scores and challenges addressed by the primary studies. The authors identified themes regarding the challenges that the studies identify and/or address independently. These themes were compared and discussed among the authors. In the case of conflicting themes, disagreements were resolved with the assistance of the third co-author. Finally, the themes were grouped according to their underlying nature into five categories: technological, human factors, organisational, communication and collaboration, and regulatory challenges. Regarding data synthesis, the previous literature has proposed the following two approaches: descriptive/narrative data synthesis and quantitative data synthesis [43]. We followed the descriptive data synthesis approach by identifying themes based on the data extracted from the selected studies. Additionally, we measured the frequencies of these themes, in order to outline dimensions for future research. It is noteworthy that these frequencies do not indicate the importance of the challenges but provide insights into research gaps. 4 RESULTS In the following sections, we provide an overview of the primary studies and findings related to our research question. Overview of primary studies In this section, we present contextual information about the primary studies. In particular, we outline the distribution of primary studies based on publication year and research methods used, publication type, and quality scores using descriptive representations, such as a simple bar chart, multiple bar chart and a pie chart. Figure 2 shows the number of primary studies, distributed based on the research method and the year of publication. It is easy to notice the dominance of experiments throughout the majority of the years. For instance, 9 out of 16 studies published in 2013 used experiments as the main research method. Overall, 29 out of 70 studies (41%) adopted only experiments as their research method and 7 out of 70 studies (10%) adopted both surveys and experiments, indicating controlled experiments as the most used research method (36 out of 70 studies). This is not surprising as experiments reduce complexity through the control of all variables other than the ones under investigation, given their reductionist nature [44]. For instance, in [16], the authors aimed to investigate t