Value‐oriented quality metrics in software development: Practical relevance from a software engineering perspective

IET SoftwareVolume 16, Issue 2 p. 167-184 ORIGINAL RESEARCH PAPEROpen Access Value-oriented quality metrics in software development: Practical relevance from a software engineering perspective Philipp Haindl, Corresponding Author Philipp Haindl philipp.haindl@jku.at philipp.haindl@scch.at orcid.org/0000-0001-6075-5286 Department of Business Informatics - Software Engineering, Johannes Kepler University, Linz, Austria Software Competence Center Hagenberg, Hagenberg, Austria Correspondence Philipp Haindl, Software Competence Center Hagenberg, Hagenberg, Austria. Email: philipp.haindl@jku.at and philipp.haindl@scch.atSearch for more papers by this authorReinhold Plösch, Reinhold Plösch Department of Business Informatics - Software Engineering, Johannes Kepler University, Linz, AustriaSearch for more papers by this author Philipp Haindl, Corresponding Author Philipp Haindl philipp.haindl@jku.at philipp.haindl@scch.at orcid.org/0000-0001-6075-5286 Department of Business Informatics - Software Engineering, Johannes Kepler University, Linz, Austria Software Competence Center Hagenberg, Hagenberg, Austria Correspondence Philipp Haindl, Software Competence Center Hagenberg, Hagenberg, Austria. Email: philipp.haindl@jku.at and philipp.haindl@scch.atSearch for more papers by this authorReinhold Plösch, Reinhold Plösch Department of Business Informatics - Software Engineering, Johannes Kepler University, Linz, AustriaSearch for more papers by this author First published: 25 November 2021 https://doi.org/10.1049/sfw2.12051Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract When following the principles of value-based software engineering, business, customer satisfaction, and engineering considerations need to be balanced to develop and operate the software so that it satisfies the different stakeholders' expectations. This, however, requires knowing the relevant quality metrics covering these value-oriented expectations and potential sources for their measurement. In this work, a categorisation of value-oriented quality metrics that are practically relevant is presented. Therefore, the authors conducted an online survey with practitioners who assessed the relevance of 61 value-oriented metrics, gathered from a preceding systematic mapping study. The authors grouped these metrics into 10 categories, based on financial, customer satisfaction, value proposition, and creation perspectives. Also, the authors examined the frequency of particular steps at which these measures accrue and identified their most relevant data sources. The participants rated metrics for feature reliability, performance, as well as test and development efficiency as most relevant for value orientation. According to the participants, the authors' collection covers all relevant metrics for addressing financial and market and feature usability aspects. The authors' categorisation and the metrics' relevance assessments shall support software engineers in selecting relevant metrics and their sources for software product development. 1 INTRODUCTION Value-based software engineering allows regarding different stakeholders' value appraisal of the developed software by evaluating development and operational metrics relevant for the stakeholders. Thereby, these metrics serve as indicators of how well specific values for the respective stakeholder are fulfilled. Appropriate quality metrics for value-based software engineering must take into account the different value considerations of stakeholders, for example, business, security, customer-related or technical considerations [1, 2]. However, existing classifications of quality metrics isolatedly focus on one single-stakeholder perspective, such as the development perspective [3-5], the business perspective for the customer value analysis [6, 7], the customer perspective [8, 9], or the perspective of feature ideation and improvement [10, 11]. The common notion of software quality metrics primarily captures quality-related aspects that contribute value for technical stakeholders, for example, maintainability or performance efficiency of the software. As such, they neglect aspects that are particularly relevant for the value appraisal of stakeholders outside the technical domain, for example, for business stakeholders who expect that the software is frequently used by customers or generates revenue aligned with the business model. As an example, after deploying a new feature of a mobile app into production, observing changes of the customers' satisfaction in app stores, app outage reports, or customer churn is paramount for deciding about required operational adjustments, bug fixes, or code improvements in subsequent development phases. However, due to the absence of a comprehensive list and classification of metrics addressing the different stakeholders' value-oriented considerations, it is cumbersome to meet these value expectations. This paper seeks to close this research gap by presenting a concrete list and categorisation of practically relevant value-oriented quality metrics. The metrics are based on our former systematic mapping study of over 140 papers [12] and a complementary narrative literature review. The relevance and completeness of these metrics were assessed based on the results from a survey with 41 practitioners who have different roles in the software product lifecycle. Also, we present the identified sources in companies for measuring these metrics in practice. The paper is divided into 10 sections, whereby the remainder of this section elaborates the research objective and contributions of this study. Section 3 describes the research methodology and particularly the questionnaire design and participant recruitment for the survey. Section 4 details the demographic aspects, professional information, and backgrounds of the participants. Following, Section 5 highlights our findings regarding the software development practice in companies. The relevance assessments of the metrics through the participants are discussed in Section 6, complemented by the categorisation of the metrics along value-oriented stakeholder perspectives [13] presented in Section 7. The results from assessing the completeness of the metrics are given in Section 8 whereby we also elaborate on the metrics declared as missing by the participants. We summarised these missing metrics through qualitatively analysing individual survey responses. In Section 9, we discuss the applicability of the results from the perspective of practitioners and researchers, respectively. Section 10 outlines the recognised threats to the validity of the presented results, before concluding our work and sketching possible directions for future work in Section 11. 1.1 Research goal and contributions The overall research goal of this paper was on gathering empirical data regarding the relevance and completeness of quality metrics for selected value considerations, identified through the preceding systematic mapping study. This research goal can be refined into the following objectives: Assessing the practical relevance of the metrics identified by our literature study, from the perspective of practitioners in the software product lifecycle. Evaluating to which degree a metric category is covered by the associated metrics. Detecting additional metrics having value for critical stakeholders in software development. Identifying suggested sources for measuring these metrics. When applying the idea of value-based software engineering [14], different value considerations from business, customer, and development stakeholders need to be addressed and specified in a concrete and unambiguous manner using quantitative quality metrics. These metrics also facilitate balancing contradictory value considerations of stakeholders and evaluating them in relation to a company's business model [15-17]. The presented paper contributes to the current research challenges for value-oriented software engineering as follows: A categorisation of value-oriented quality metrics, whose necessity was identified through empirical case studies and the lack of the same criticised by several authors [5, 18, 19]. Empirical findings regarding the practical relevance of value-oriented quality metrics from the perspectives of software engineering practitioners. The authors of Refs. [15, 17], thereby, especially highlighted roles such as product and project management, software development, and operation management as important. A summary of the most relevant measurement sources to operationalise these metrics in practice. Knowing these measurement sources is the prerequisite to developing appropriate software tools [20, 21] that automatically acquire and evaluate these metrics. 2 RELATED WORK In a qualitative multiple-case study [18], practitioners were interviewed about the practical challenges of DevOps adoption in companies. The respondents mentioned that common classifications of software quality metrics primarily focus on software product quality, thus neglecting metrics focussing on the process of software development, delivery, and operation themselves. Similarly, the authors in Ref. [19] conducted an exploratory case study interviewing several software engineers about quality-related requirements in DevOps. The interviews unveiled that in their projects the experts often implicitly are confronted with non-technical requirements such as, for example, the alignment of the software with the business model, regulatory considerations or the demand for user-interaction monitoring for data-driven software improvement based on feature usage. While these types of requirements explicitly are articulated by stakeholders as non-functional requirements and, thus, are part of their value appraisal of the software, they do not adhere to the technical notion of metrics underlying, for example, the ISO/IEC 25010 specification [22]. As a result, they are often not clearly specified using metrics and appropriately monitored during software development. Instead, the fulfilment of these stakeholder requirements is analysed qualitatively ex-post and often outside the software engineering context. The necessity for a categorisation of relevant metrics for value-based software engineering can also be inferred from the number of studies elaborating on the challenges arising from the misbalance of these metrics. For instance, the requirement to collect operational usage data for A/B testing [11, 23] aims on novel feature ideation to improve the value of the developed software for the customer. While the authors stress the importance of these requirements and also their heterogeneity due to the different value expectations of stakeholders, they do not present concrete metrics that allow us to quantify and monitor the fulfilment of these requirements. As an example, the customer-focussed value consideration sketched above not only requires a technical implementation for operationalising the respective data but also an overall product strategy to also take into account business-focussed value considerations and evaluate the fulfilment of these considerations suitably. Also, maintainability requirements towards novel software features might be higher than of features having less strategic importance for a company. In summary, these works stress that in value-driven software engineering often contradictory requirements from stakeholders of different domains need to be fulfilled. Balancing these requirements presupposes defining concrete assessable fulfilment criteria based on value-oriented metrics and categories. Categorising metrics by their spanning theme could also help in balancing these contradictions in a more structured manner. To address this challenge and to identify the metrics relevant for the stakeholders' value appraisal of software, we conducted a systematic mapping study about NFRs in DevOps [12]. Thereby, we carved out recurrent focus areas and themes of NFRs in DevOps that reflect relevant value considerations of stakeholders and derived quality metrics thereof. One important finding of this study was that only a few studies present metrics whose relevance in practice is also empirically validated. The monitoring of stakeholder objectives outside the technical domain, addressing for instance market penetration, customer satisfaction or regulatory compliance, is still hindered by the absence of a common collection of metrics for these objectives. Despite the large number of metrics mentioned in the examined papers, only a few of these metrics are empirically validated regarding their practical relevance to reflect the business, strategic, organisational or customer value of software features. Thus, the question of practical relevance of these metrics can, however, only meaningfully be answered based on an empirical investigation. 3 RESEARCH METHODOLOGY The foundation for this survey was the value considerations identified through the systematic mapping study. As already pointed out, one important finding of this mapping study was the apparent lack of empirical studies on the practical relevance of the value considerations described in the studies as well as appropriate indicators for evaluating the fulfilment of these value considerations. In this context, value considerations reflect important aspects of the stakeholders' value appraisal while value criteria describe concrete indicators relevant for stakeholders to judge whether these value considerations are fulfilled. As a starting point for the survey, we defined a compact set of metrics for these value criteria that can be assessed by practitioners and also used to identify potential measurement sources. Thereby, we included studies returned from the mapping study, which elaborate on the practical applicability of the herein described value criteria and addressed stakeholder objectives. Additionally, we conducted a narrative literature review to find further relevant studies giving indications for concrete value criteria. Afterwards, we grouped the resulting set of metrics into four main categories and iteratively refined them into 10 categories. For the sake of the compactness of this paper, we provide the literature references for these metrics as an online resource (https://bit.ly/3jXf2Uf). The aim of this study was to assess the relevance of these metrics particularly from the perspective of practitioners such as software engineers, product and project managers, software testers and operational staff, as they pursue different value considerations in software engineering. Therefore, we developed an anonymous online survey targeting these stakeholders, which could be completed within about 20–25 min and distributed via professional social networks and personal invitations. 3.1 Questionnaire design The questionnaire for the survey consisted of 72 questions, grouped into five question blocks, and an end page and was developed based on the guidelines of Kitchenham [24]. The details of these question blocks are as follows: Professional Information: In the first block, the participants were asked about their current affiliation, the type of software they are developing/operating, and the industry sector of their division/company. Roles and Experience: In this block, the participants had to select three roles that they currently have or previously had in software engineering projects (potentially also in different companies). At least one role had to be given by each participant. For each role, the participants were asked to also give their years of experience. Software Development Background: The third block gathered information pertaining to the frequency of particular software engineering steps and the typical length of development iterations in the companies. Knowing about the frequencies of these steps is important for operationalising the examined metrics in practice. Each metric has a particular point in time or step in the development lifecycle, where it can be measured. Metric Assessment: To not overburden the participants and, in the worst case, cause them to quit the survey or make unreflected assessments, the survey showed each metric category on an individual page. The idea behind this was to keep participants motivated, as they could decide after each block whether they wanted to answer another one. Furthermore, this should ensure that each participant assesses at least one block of metrics. The sequence of metric blocks was randomly chosen by the survey for each participant. In each metric assessment block, the participants assessed the relevance of five to eight metrics on a five-point numerical scale from one (low relevance) to five (high relevance). At the beginning of each question block, the survey showed an individual description (in two or three sentences) of the corresponding metric category to the participants. When describing the rationale of a metric category, we intentionally used basic terminology to ensure a common understanding of the category for all participants regardless of their role in the software development process. For each metric, the participants could also state that they do not know the metric and, thus, are unable to rate it. The answers to the metric assessment blocks were mandatory, meaning that participants either needed to rate all metrics or state that they do not know an individual metric. Afterwards, the participants were asked if the presented metrics completely cover the value considerations and, thus, the criteria underlying these considerations. The answer to this question was mandatory. In case of the answer incomplete, the participant was asked to describe missing metrics in a free text field. Eventually, the participants were asked to textually describe possible data sources for acquiring measurements for these metrics. Answering this question could optionally be skipped. At the end of each assessment block, the participants were asked whether they wanted to continue with another block or quit the survey. At that point, the survey showed information to the participants to explicitly express appreciation for the already spent effort and emphasised the importance for continuing assessing another set of metrics. By giving participants the freedom to leave the survey after each assessment block, the participants should be kept motivated and prevented from unthinkingly rating the metrics just to quickly finish the survey. End of Survey: On the final page, the survey again showed appreciation for the participant's time and asked whether the participant was interested in getting the final results of the survey by email. To not relate this email address with a particular participant, a link for sending an email with a given subject was displayed. 3.2 Participant recruitment Two approaches were followed for recruiting participants for the survey. First, by personally inviting professionals who are knowledgeable in the examined topics and, second, by promoting the survey in expert groups of professional networks such as XING (http://www.xing.com), ResearchGate (http://www.researchgate.com), and LinkedIn (https://www.linkedin.com). When personally inviting people, we always outlined the general objective of the survey and provided an access token to the survey. The access token allowed to monitor which groups of participants (associated with this token) already had answered the survey and also prevented unsolicited ratings from external people. In the personal invitation email, we also asked to distribute the link to the survey among colleagues to additionally increase participation in the survey. For recruiting participants on the mentioned professional networks, we approached only people in expert groups that deal with the examined topics. Therefore, we described the general objective of the survey and provided the link to it as well as the access token to monitor participation and, in case of too few respondents, repeated my call for participation. In this manner, we approached 13 topic-related expert groups on LinkedIn, one in ResearchGate, three on LinkedIn, and sent personal invitations to seven groups of professionals that we personally know. 4 DEMOGRAPHIC ASPECTS OF PARTICIPANTS The survey was provided in the time frame between June until October 2020. In total, 81 people started the survey either by following the personal invitation link or by clicking the link in any of the expert groups in the professional social networks. Afterwards, 31 participants left the survey right before answering the first question block, resulting in 60 participants who answered the professional information block pertaining to their affiliations. Thereafter, nine participants quit the survey and the remaining 51 participants proceeded to the page asking them about their roles and experiences in their professional lives. After answering about their roles and experiences, four participants dropped out before being asked about the software development background in their companies, and six further participants quit the survey after this question block. The remaining 41 participants successfully finished the survey, meaning that they assessed at least one block of metrics. Hence, the percentages given in this presentation relate to these 41 participants as they not only completed at least one metric assessment block but also provided enough information for an appropriate analysis in the previous question blocks. In the median, the participants answered four question blocks. 4.1 Professional information In the first question block of the survey, the participants gave their current affiliation and the industry sector of their companies. The majority of participants (49%) works in large companies with more than 1000 employees, followed by 24% from companies with less than 1000 employees, 14% from companies with less than 250 employees, 6% from companies with less than 100 employees, and 4% from companies with less than 50 employees. Summing up all participants from companies with 50 to 1000 employees, this group accounts for 96% of the participants. Only two percent of participants originate from academic organisations or are self-employed. As the survey explicitly aimed at gathering assessments regarding the relevance of the metrics from practitioners, this distribution also supports this objective. From the perspective of the companies' industry sectors, the vast majority of participants (49%) works in companies in the field of professional IT and software services, 14% in companies that develop or operate software for production and manufacturing, 10% for public administration, 8% for commerce, 6% for telecommunications, 6% for transportation/logistics, and equally 2% in health and education services. Four percent of participants work in other industry sectors. The survey did not allow participants to textually describe alternative industry sectors. As a result, no statement can be given about the four percent of participants having chosen other as their companies' industry sector. 4.2 Background and experience The second question block examined the background of the participants, meaning their roles and experience in these roles, potentially also in previous affiliations. Figure 1 shows the distribution of these roles in percent underneath the corresponding bar, whereby the largest shares can be attributed to software engineering roles such as software engineers (20%) and architects (17%), totalling both to 37%. Summing up the less technical roles such as consultants, product owners, project, and product managers totals to 21%. Roles having a focus on the operation of the software can be seen by infrastructure engineers, operations managers, and site reliability engineers. These roles are held by 22% of participants. Although the participants of the survey have some relation to business functions and operations, we did not achieve full coverage of all roles necessary for value orientation in software development. Therefore, the findings of this survey reflect the view of more technical participants onto value orientation but lacks especially the business-oriented side. FIGURE 1Open in figure viewerPowerPoint Average experience and distribution of job roles For each role, the participants also had to give the number of years in experience in these roles (cf. Figure 1). Interestingly, the participants have the most experience as consultants and operations managers. Thereby, it has to be noted that in the field of IT consultancy, companies often refer to software engineers as consultants. However, the survey did not further examine the duties associated with these job roles and possible similarities in the companies. In summary, the distribution shows that the participants on average have more years of experience in technical domains than in fields such as project or product management. Also, the participants were asked which type of software product they are developing or operating. The analysis of the participants' answers to this question highlights that the majority of 45% of participants' works in the context of business information systems, 18% work on development tools, 16% on embedded systems, 10% on knowledge-based expert systems, 6% on mobile systems, and 2% on service-oriented systems. Four percent work on software outside the given fields, for example, mainframe systems. It can be depicted from these numbers that development tools and embedded systems are the two second important types of software that the participants work with. 5 SOFTWARE DEVELOPMENT PRACTICE IN COMPANIES The third question block examined particular aspects of the software development practice in the companies, such as the frequency of particular engineering steps during software development. When answering these questions, participants could also indicate not knowing the frequency of individual steps. Figure 2 shows the frequency of the different steps, as known to the participants. In summary, most of the examined steps are executed on a frequent basis mostly hourly or daily in the companies. Interestingly, steps such as evaluating software security or quality reports, or performing static code analysis are never conducted in around 19% of the companies. Also when it comes to testing, around 14% of companies never conduct concrete testing steps in their development lifecycle. As expected, steps targeting developing and deploying software in many companies are executed in shorter intervals than steps targeting on evaluating software architecture, quality or security reports. FIGURE 2Open in figure viewerPowerPoint Software development steps in the companies Asked about the length of the development iterations in their companies, the participants mentioned weeks (87%) and months (11%) as typical durations. Two percent of participants did not know the duration of iterations in their companies. 6 RELEVANCE ASSESSMENT OF METRICS Based on the assessments from 41 participants we calculated the average assessment for each of the 61 metrics. The average assessment was calculated from the individual relevance assessments ranging from 1 point (very low) to 5 points (very high) of the participants' responses. For getting a quick overview of the most relevant metrics, we ordered them by average assessment and standard deviation in decreasing order. Thereby, we regarded the practical relevance of metrics having an average assessment above 4.5 points as very high, an average assessment between (including) 3.5 and 4.5 points as high, between (including) 2.5 and 3.5 points as moderate, between (including) 1.5 and 2.5 as low, and below 1.5 as very low. Accordingly, the participants considered 23 metrics as highly relevant, 37 metrics as moderately relevant and one metric as little relevant in practice. In addition, we calculated the standard deviation and skew of the individual assessments and depict the details for each metric in Table 1. The standard deviations of all metrics range between 0.94 and 1.61. A low standard deviation among assessment values can be interpreted as consensus among the participants regarding the relevance of metrics, whereas contrarily high values can be interpreted as disagreement. TABLE 1. Relevance ranking of metrics Metric Average Relevance Standard Deviation/Skew (*-1) Availability 4.16 High 1.40 / 0.97 Downtime 4.11 High 1.33 / 0.89 User count 4.00 High 1.26 / 0.93 Response time 3.94 High 1.41 / 0.65 Build duration 3.93 High 1.36 / 0.71 CPU usage 3.88 High 1.20 / 0.50 Feature test coverage 3.88 High 1.29 / 0.58 Unit test coverage 3.88 High 1.38 / 0.49 # Missed quality gates 3.83 High 1.31 / 0.71 Automated system tests ratio 3.80 High 1.26 / 0.84 # Reliability incidents 3.78 High 1.23 / 0.22 # Vulnerabilities 3.72 High 1.19 / 0.91 Complexity of architectural constraints 3.71 High 1.10 / 0.76 Complexity of features 3.71 High 1.16 / 0.81 Memory consumption 3.71 High 1.21 / 0.65 # Deployments to QA systems 3.62 High 1.39 / 0.15 Security rating 3.61 High 1.20 / 0.44 # Completed change requests 3.58 High 1.20 / 0.29 Technical debt 3.58 High 1.22 / 0.44 Customer satisfaction index 3.53 High 1.41 / 0.49 System test duration 3.50 High