dataesr/scientific-paragraphs-categorization huggingface dataset:
I. INTRODUCTION Nowadays, a company's ability to innovate is increasingly driven by software. Digital technologies play a key role in the transformation of many industrial companies  [1] , especially in sectors like the automotive industry, where software (together with electronics) is responsible for over 90% of all innovations  [2] ,  [3]. In this context, software quality makes the difference and is an essential competitive factor for company success. Companies use modern scalable data ingestion technologies (e. g. , Apache Kafka 1  , ActiveMQ 2  ) together with data visualization and analytics technologies (e. g. , Tableau  3  , Microsoft Power BI  4  ) to learn more about their businesses  [4]. These technologies have enabled the concept of the real-time enterprise, which uses up-to-date information and acts on events as they happen  [5].
g. , product quality, product readiness, and process performance) in software analytics tools. We propose using the well-known concept of Quality Model (QM)  [13] -  [16]  in software analytics tools as the instrument to bridge the gap between low-level quality concepts related to development and usage and high-level quality-related strategic indicators. Therefore, the goal of this research is to understand practitioners' perception regarding the integration of QMs in software analytics tools in ASD companies in order to effectively assess and improve quality-related strategic indicators. Since we address the perception of practitioners, our aim was to conduct an empirical study in industry. To this end, we needed a tool implementing the concept we wanted to evaluate. Given that no such tool is freely available on the market, we opted for the Q-Rapids tool.
The Q-Rapids tool, developed as part of the Q-Rapids project  [17] , offers software analytics capabilities that integrate QMs to assess and improve software quality in the context of ASD. Its main functionalities are: (a) real-time gathering of various types of data related to the development and usage of a software system, which are the input for a QM; (b) real-time modeling of this data in terms of a QM in order to reason about aggregated quality-related strategic indicators; (c) presenting this QM and data to decision makers in a multi-dimensional and navigational dashboard for use during ASD events, such as sprint planning or daily stand-up meetings. In order to understand the practitioners' view regarding our goal, we conducted a case study across four companies, involving 26 practitioners, to investigate the following key aspects related to the Q-Rapids tool: • Its understandability, reliability, and usefulness to assess software quality; • Its level of detail, relevance, perceived usefulness, and behavioral intention to improve software quality; • Its challenges and lessons learned, as seen by adopters in realistic settings.
This paper is structured as follows. Section II presents the background and related work. Section III presents the object of study in this case study across four companies: the integration of a QM within a software analytics tool (i. e. , the Q-Rapids tool). Moreover, it provides details on how the tool can be used to assess and improve software quality. Section IV describes the research methodology of our case study. Section V presents the results with respect to the participants' perceptions on exploring the tool to assess and improve software quality. Section VI discusses challenges and lessons learned as seen by adopters in ASD settings using the Q-Rapids tool. Section VII reports the limitations of this work. Finally, Section VIII concludes the article and presents future work.
A. BACKGROUND Quality is defined by ISO 8042  [18]  as "the totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs". This definition is too abstract to be operationalized directly, and is one of the reasons why there has been a multitude of software QMs proposed in the last four decades (e. g. ,  [19] -  [23] ) that refine high-level concepts of "quality" (like reliability or efficiency) down to the level of metrics (as number of bugs or response time). One popular example widely adopted in industry is the ISO/IEC 25010 standard  [14] , which determines the quality aspects to be taken into account when evaluating the properties of a software product. Two more recent examples well known in industry are Quamoco and SQALE  [24]. Quamoco  [16]  is a QM integrating abstract quality aspects and concrete quality measurements.
SQALE  [25]  computes technical debt indexes based on the violations of quality rules in the source code. Other works propose refactoring models to improve a particular quality aspect of the existing model or code, and different extensions to the traditional concept of quality (such as incorporation of non-technical criteria into ISO/IEC 9126-1 QM  [26] ). Software analytics is defined as "analytics on software data for managers and software engineers with the aim of empowering software development individuals and teams to gain and share insight from their data to make better decisions"  [6]. In this context, analytics results should include some actionable knowledge that can be used to improve software quality. Software analytics is used in various scenarios to assess concrete problems, e.
g. , use of process data to predict overall project effort  [27] , use of security data to identify indicators for software vulnerabilities  [28] , or classification of app reviews  [29]. In this context, software analytics tools provide features for specifically visualizing software development aspects. In our view, some notable examples are SonarQube 11  , Kiuwan, Bitergia  12  , and Tasktop, all of which have been discussed in related work. ASD relies on short feedback cycles as a way to provide flexibility and rapid adaptation to market fluctuations. In this context, decisions are also made more frequently. For instance, product releases and the related decisions take place in short intervals, instead of months/years as was the case with traditional software development approaches such as the waterfall model.
Popular agile practices such as continuous integration  [30]  provide a tight connection to development to ensure errors are detected and fixed as soon as possible. The current tendency to shorten feedback cycles even further (e. g. , continuously getting feedback from customers and being able to react on that) enhances the potential for software analytics. Continuous software engineering  [31] , which has its roots in ASD, represents a holistic approach to providing tight connections among all software development activities, including not only integration but also aspects such as business and development (BizDez) and development and operations (DevOps). Our work aims to further understand the challenges limiting industrial uptake of QM-based software analytics tools in ASD companies, as seen by practitioners.
B. RELATED WORK Software analytics plays a major role in ASD and continuous software engineering since, properly used, the enormous amounts of data from different systems (e. g. , continuous integration system, management tools, etc. ) have proved increasingly useful for guiding (almost) real-time decisionmaking  [32] -  [34]. Indeed, companies like Microsoft are hiring data scientists for their software teams  [35]. ASD and continuous software engineering have created numerous new opportunities for observing user behavior and monitoring how systems and services are being used; for identifying unexpected patterns and runtime issues; for monitoring system quality attributes; and for collecting real-time data to feed both business and technical planning  [36] ,  [37]. The main objective is to constantly monitor and measure both business indicators and infrastructure-related metrics in order to facilitate and improve business and technical decisionmaking  [38].
In the specific case of software quality, continuous quality monitoring enables transparency to foresee, trace, and understand important aspects of product quality in real time  [38] ,  [39]. Support for product quality management is particularly relevant in ASD, since it tends to overlook quality aspects in favor of product functionality  [17]. Although the literature reports on promising advances regarding the use of analytics in continuous software engineering  [32] -  [34] , many challenges still exist, with the lack of software analytics tools being among the most relevant ones  [33]. When we look for software analytics tools typically used in ASD, we find a large number of commercial and academic software analytics tools that are available on the market. Examples of commercial tools focusing on concrete quality aspect are SonarQube, Kiuwan, Bitergia, New Relic 13  , Datadog 14 , and Taktop.
SonarQube focuses on continuous code quality based on static code analysis, including assessment of code smells, bugs, and vulnerabilities. Kiuwan products focus on the detection of code security vulnerabilities, also offering a tool specifically for Open Source Software (OSS) risk analysis. Bitergia provides actionable and customizable dashboards for analyzing OSS. New Relic allows developers to install monitoring agents and gather real-time insights from users such as software failures and performance improvements. Datadog monitors cloudscale applications; provides monitoring of servers, databases, tools, and services; and supports the full DevOps stack. Tasktop aims to integrate and coordinate value streams across the DevOps stack. In addition to these commercial software analytics tools, it is worth mentioning value stream management tools for capturing, visualizing, and analyzing critical indicators related to software product development  [12].
Some of these tools even strongly support the creation of new dashboards from data and advanced data analytics (e. g. , machine learning). With respect to tools in academia, we find CodeFeedr  [40] , Perceval  [41] , and SQuAVisiT  [42]. CodeFeedr is a real-time query engine. Perceval performs automatic and incremental data gathering from almost any tool related to open source development, which can be used in Bitergia dashboards. SQuAVisiT is a generic framework supporting maintainability assessment. Based on the aforementioned software analytics tools, we can see that the data analytics trend has had a great impact on software engineering as well (i. e. , software analytics), particularly in the short feedback cycles of ASD and continuous software engineering. However, application of QMs in the software analytics tools used in industry is not widespread, despite the need for "a view into planning, health indicators, and analytics, helping them collaborate more effectively to reduce waste and focus on work that delivers value to the customer and the business"  [12].
A notable exception is the SQALE QM within SonarQube. However, its limitation is that its main functionality of static analysis of the source code is limited to a single data source (i. e. , the source code). We can conclude that the aforementioned software analytics tools do not offer full capabilities, including an integrated QM, to provide qualityrelated strategic indicators by using software analytics results. Therefore, at the beginning of the Q-Rapids project, we decided to build the Q-Rapids tool, which integrates highly customizable (instead of pre-defined) QMs. Some researchers have investigated the success factors as well as the needs and challenges related to the application of software analytics in industry  [6] ,  [43]. Huijgens et al. conducted an empirical study to identify success and obstruction factors regarding the implementation of software analytics in the context of continuous delivery as a service (e.
A. THE Q-RAPIDS QUALITY MODEL Both academic and industry partners of the Q-Rapids project have iteratively created a QM for software analytics, whose main characteristic is that it offers tool-supported customization (integration into software analytics tools) to define strategic indicators related to the quality of software product and development processes based on company needs. This QM aims at enabling decision makers to improve identified quality deficiencies. In the following, we will present the elements of the QM and how these elements are computed from automatically ingested raw data. For details on the initial creation and previous iterations, the reader is referred to  [44] ,  [45]. Following common approaches, we defined the QM for the Q-Rapids tool as a hierarchical structure composed of five types of entities, each of them serving a well-defined purpose (see Figure  1 ): • Strategic Indicator -Quality-related aspect that a company considers relevant for its decision-making processes.
B. THE Q-RAPIDS SOFTWARE ANALYTICS TOOL The Q-Rapids tool  15  provides continuous assessment, which could be real-time, of the strategic indicators to decision makers based on the Q-Rapids QM. Figure  2  shows an excerpt of the conceptual architecture of the Q-Rapids tool, depicting its modules related to data analytics capabilities and its data flow, which adopts the idea of the lambda architecture approach used for Big Data solutions  [46]. The main modules of the tool are Data Gathering, Data Modeling and Analysis, and Strategic Decision Making. The Data Gathering module is composed of different Apache Kafka connectors to gather data from heterogeneous external data sources, such as static code analysis (e. g. , SonarQube), continuous integration tools (e. g. , Jenkins), code repositories (e.
g. , SVN, Git, GitLab), issue tracking tools (e. g. , Redmine, GitLab, JIRA, Mantis), and software usage logs. The Data Modeling and Analysis module uses the gathered data and the QM definition to assess software quality. The QM is highly customizable to support any company's needs. For instance, users of the Q-Rapids tool can define new strategic indicators, product/process factors, assessed metrics, and their relationships, as well as the frequency of execution of the QM assessment (e. g. , daily, hourly). Another example: Since the QM works as a plug-in, it can also be defined from data mining and analysis. The Strategic Decision Making module is responsible for the end-user interface and provides two different types of dashboards: (a) the Strategic Dashboard, providing homogeneous visualization for assessing the higher-level elements of the QM; and (b) the Raw Data Dashboards, providing specific visualizations for the data gathered from the source tools.
C. THE Q-RAPIDS TOOL FOR ASSESSING AND IMPROVING SOFTWARE QUALITY This section describes two scenarios where the Q-Rapids tool can be used by decision makers to assess or improve the quality of their products. We follow the definitions of "assessing" and "improving" software quality specified by Kläs et al. [23]. Assessment refers to the process in which "a concept is quantified, measured and compared to defined evaluation criteria to check the fulfillment of the criteria"  [23]. Improvement refers to the process in which "relevant factors (i. e. , variation factors) influencing the concept of interest are known. Therefore, the concept of interest can be improved by improving its influencing factors"  [23].
1) ASSESSMENT SCENARIO WITH Q-RAPIDS The (infamous) ACME company needs to deliver the next release of one of their higher-quality products, NeverLate, on time. Therefore, Bob, the product manager in charge of NeverLate, decides to use the Q-Rapids tool in order to assess and monitor Blocking situations  16. At the beginning of the sprint, Bob receives an alert because the Blocking strategic indicator has dipped below some predefined threshold. The Q-Rapids tools visualizes this event by changing the traffic light related to this indicator from green (good quality) to orange (neutral) in the Strategic Indicators View (Figure  3 , needle in the radar chart on the left). He delves deeper into the QM to analyze the situation. Taking up the QM, the Blocking strategic indicator is impacted by two product factors: Blocking code and Testing status.
A. RESEARCH SETTING Our research context consisted of four pilot projects in the four companies participating in the Q-Rapids project. As reported in Table  II , the companies ranged from small and medium-sized enterprises with up to ten products on the market to very large corporations with more than 1,000 products on the market. They develop products using ASD for various application domains such as telecommunications, security, military, transport, health, and public administration. The companies' pilot projects had three essential things in common that made it possible to run a case study  [47]  across them: They had an ASD (Scrum-like) way of working; the stakeholders (i. e. , product owners, managers, and developers) were interested in having tool support to assess and improve software quality in their setting; and the Q-Rapids tool presented in Section III was deployed and used by these stakeholders in such settings.
In the following, we will briefly describe the context of the four use cases. One common characteristic is that the four companies had already used Q-Rapids in earlier pilot projects than the ones evaluated in this case study (i. e. , UC1, UC2, UC3, and UC4 from Table  II ). UC1. Company 1 used Q-Rapids in its main product Modelio 3. 8 and in Modelio NG (UC1), which is a customization for a customer. Modelio is the latest generation of a 25-year-old product line of a model-driven tool suite dedicated to expressing and managing requirements, modeling software architectures, building accurate UML models, generating a full range of documentation, and automating application code production for several languages. In UC1, Company 1 used Q-Rapids with the aim of improving the quality of Modelio NG by leveraging QRs during the development phase of new versions of the product line.
e. , UC2). In UC2, Company 2 decided to use their expert knowledge to identify interesting strategic indicators not provided by other tools to be presented from different viewpoints by the Q-Rapids tool. The reported challenges and lessons learned are based mainly on the deployment experience with UC2. UC3. Company 3 implemented the solution in two contexts. For the first context, the focus was on the use of a proof of concept of the Q-Rapids solution by a production test software team in the public safety domain. The research partners facilitated or collaborated on multiple sessions to determine the strategic indicators as well as process factors and metrics. This included the company's internal development of the software lifecycle development process as well as the identification of necessary data sources and alignment of the development tool chain with the Q-Rapids solution.
D. DATA COLLECTION INSTRUMENTS To support the aforementioned evaluation design, the researchers used the following six instruments (available in the Appendix) during the evaluation: 1. Slides with evaluation goals, agenda, and the informed consent and demographics forms. 2. Scripts for a live demo of the Q-Rapids tool assessment and improvement scenarios to give researchers a common artifact to show to the users of the Q-Rapids tool (similar to the examples in Section III). In addition, a document comprising the specification of the QM (e. g. , strategic indicators). 3. The task description, consisting of specific scenarios for assessing and improving the software quality of the products in which the participants were involved on a daily basis. 4. A questionnaire, to collect the practitioners' perception regarding the use of the Q-Rapids tool, using the constructs of Table  III.
E. DATA ANALYSIS The experimenter and the observer transcribed the participants' answers (regarding the tasks, the questionnaires, and the cards from the debriefing sessions) and the observation protocol into three standardized Excel templates. This served to keep the data analysis consistent among the four companies. We first carried out within-case analyses of the quantitative and qualitative data collected for each company. Then we compared, correlated, and integrated the results among the companies (cross-case analyses)  [55]. Quantitative analysis. We report descriptive statistics including the sample size (N), minimum (Min), maximum (Max), median (Mdn), and modal value (Mode) for the quantitative analyses. If a participant answered more than half of the questions of one construct with "I don't know" (see the questionnaire in the fourth item of Section IV.
Therefore, we also report the significance levels p and the standardized test statistics of the One-Sample Wilcoxon Signed Ranks Test (see Table  V ). We used IBM SPSS Statistics 19 (including IBM SPSS Exact Tests for analyzing small samples) and set the confidence level of the test at 95% (i. e. , α = 0. 05). Qualitative analysis. We used thematic analysis  [57]  to analyze the participants' feedback on the Q-Rapids tool. At least two researchers derived themes inductively by coding and interpreting all observation protocols, independent of each other. Then these researchers compared their results and resolved any deviations. Moreover, we performed several peer review meetings, which included all experimenters, observers, and analysts, to review the interpretations of the elicited qualitative data.
A. CHALLENGES REGARDING THE USE OF THE Q-RAPIDS TOOL Challenge #1. Need for tailoring to the company. A software analytics tool like the Q-Rapids tool requires a comprehensive understanding of the company's software development or lifecycle process to be able to extract a solid QM as required by the tool. Although the tool can be used from the beginning with common strategic indicators, it requires extra effort to completely adapt its QM to the needs of each company's business. Challenge #2. Need for a shared language. One aspect experienced by companies to date is the need to use a language already known and understood by the potential users. Here, the QM (names for metrics, factors, strategic indicators) has to be clear and based on the practical language known and used by product owners and developers.
Of course, this will improve with time, although it also requires a customization process for each company (the language and terms might not be standardized across all software houses in Europe). Challenge #3. Need to be informative for the user. A challenge with tools such as the Q-Rapids tool is how to interpret the information that is rendered. From an enduser's perspective, they need to understand whether a certain value of an indicator/factor is good or bad, and how it relates to the previous stage in the project or to other projects (a benchmark needs to be established and known to the users, and the 'delta' needs to be shown clearly). Challenge #4. Need for integration with other tools. Currently, in many software development companies and software houses, teams are already using some common tools (such as Jira, GitLab, and SonarQube) to manage the development process and the quality of their code and products.
Such tools are now embedded into developers' way of working, and into the processes used in companies. Therefore, when introducing the Q-Rapids tool, it has to be clear when it is going to be used to assess and improve software quality, e. g. , before checking the status of the project in GitLab or Jira or afterwards. It needs to be clarified when exactly in the agile (or any other) process this new tool should be used. Challenge #5. Need for transparency and more clarity on actions. While automatic data gathering from data sources can be seen as an advantage, some end-users (i. e. , product owners) may have a need to understand from which raw data the factors and indicators are computed and how. For instance, for one of the companies, this was the case for the product quality indicator: What data is aggregated to arrive at a single value? One of the crucial aspects for future adoption is the clarity regarding possible decisions and courses of actions.