dataesr/scientific-paragraphs-categorization huggingface dataset:
The order of magnitude the L1-dcache-load-misses metric are similar, which can explain why the execution times remain more or less the same without and with vCPUs pinning. 2) ARM: To see if the behaviour of the programs with and without pinning is only specific to RISC-V or is generalized to other architectures, we experimented with the ARM implementation of QEMU, with n c = n t and without thread affinity. QEMU ARM with machine virt limits the number of cores so we were able to run it up to only 96 vCPUs. As in RISC-V, each program is run 10 times. Additionally, we were able to run the program barnes in ARM unlike in RISC-V. We decided to put it as information in our QEMU ARM graphs. Figures  16  and 17  report T full on the benchmark subset for QEMU ARM n c = n t without and with vCPUs pinning.
As with QEMU RISC-V, T full decreases constantly as the number of threads is between 1 and 32 and begins increasing beyond 32 threads. Figure  18  shows the comparison between QEMU ARM without and with vCPUs pinning. One can see that there is no big difference, the behaviour is the same without and with pinning. We can do the same observation than in RISC-V: pinning vCPUs does not improve the execution time of our program subset. Table  II  presents the results of Perf. When comparing the number of CPU migrations for QEMU ARM without vs with vCPUs pinning, we again notice that overall, we have much less migrations with pinning, but as in RISC-V, this doesn't lead to significantly lower L1-dcache-load-misses. 3) Analysis: The metrics given by Perf show an improvement when pinning vCPUs. The number of CPU migrations is reduced but the number of L1-dcache-load-misses is close without and with vCPUs pinning.
VI. Conclusion Simulation is often useful during the design, implementation and even use of processor centric systems. The wide adoption of the RISC-V ISA and its use in current and future multi and manycore systems makes the evaluation of QEMU, the most stable and used parallel processor emulator, a lively topic. In this work, we have simulated shared-memory target systems with up to 128 processors on a 16-core/32-thread host, using Linux SMP and the PARSEC benchmark. As the result of this comprehensive experimental study, we observed that QEMU  scalability is rather good, which makes it suitable for supporting the software development of, in particular, new RISC-V based platforms. We added in QEMU the ability to pin vCPUs to host processors, and much to our disappointment, forcing the affinity did not improve performances.
Method: We have developed a novel method called SESSI (Specification and Estimation of Software Strategic Indicators) that was articulated from industrial 25 experiences with Nokia, Bittium, Softeam and iTTi in the context of Q-Rapids European project following a design science approach. As part of the industrial summative evaluation, we performed the first case study focused on the application of the method. Results: We detail the phases and steps of the SESSI method and illustrate its 30 application in the development of ModelioNG, a software product of Modeliosoft development firm.
Strategic and tactical decisions are taken by management-related roles such as product owners or project managers to reach the organizations' business goals and objectives or to manage project resources, respectively  [2, 3]. Project and product management tools (e. g. , SonarQube 1  , Kiuwan 2  ) are commonly used by development teams to exploit software development data to support operational decisions, and in some cases, tactical decisions  [4, 5]. For example, SonarQube can provide continuous code quality assessment based on static code analysis and software metrics (e. g. code smells, number of bugs and vulnerabilities). General approaches and business intelligence solutions (e. g. , Tableau  3  , Power BI  4  ) exist to extract insights from corporate data for supporting strategic and tactical decisions.
Although there exist data-and expert-driven proposals, most of them hardly consider their customization to other contexts as well as the automatic or semi-automatic data collection support for putting them forward in an organization. As a result, existing proposals are rarely used in the industrial practice. In this context, we present a novel method called SESSI (Specification and Estimation of Software Strategic Indicators) that was articulated from industrial experiences with Nokia, Bittium, Softeam and iTTi in the context of Q-Rapids European project following a design science approach. The goal of the method is to support software-development intensive organizations with guidance and tools for exploiting software development related data and expert knowledge. Specifically, the method supports the specification and assessment of software strategic indicators (SSIs).
2) To illustrate the application of the method in the context of ModelioNG, a software product of Modeliosoft, a software development firm. The remainder of this paper is organized as follows: Section 2 describes the research context in which this work has been carried out. Section 3 provides an overview of the related work. Section 4 gives a brief background on Bayesian networks, as they are used to build the estimation models proposed by the SESSI method. Section 5 provides an overview of the SESSI method detailing its phases and steps. Section 6 illustrates the application of the method in Modeliosoft, tackled as a case study and discusses relevant threats to validity that promote the correct interpretation of the case study results. Finally, Section 7 summarizes the lessons learnt on the application of the method and future work.
The goal of the industrial summative evaluation plan of SESSI is: to gain insights on the application of the method in different industrial contexts and to assess its potential worthiness. In this paper we present a summary of the results obtained from Stage 1 and focus on detailing the SESSI method resulting from the Stage 2 and the initial results of the Stage 3. Specifically, we report the phases and steps of the SESSI method and illustrate its application in the context of ModelioNG case study, a software development project of Modeliosoft.
Tool support for assessment model creation and deployment. Building assessment models for software-related indicators is a demanding process that requires supporting tools. On the one hand, we found that most proposals do not provide tools for supporting the assessment model construction process. On the other hand, most proposals also fail on providing data collectors that allow feeding the assessment model for enabling its monitoring capabilities. This makes the industrial uptake of these proposals hard. Some works, such as  [7, 9, 21, 34, 46] , provide software tools to assist some specific tasks for the assessment model creation. Some popular software tools such as source code management (e. g. Git, Subversion), source code analysis (e. g. Sonar, StyleCop) or project management (e. g. Redmine, Jira), among others, automatically collect data to provide specific metrics for software development lifecycle.
These metrics can be visually consulted by their included dashboards (e. g. SonarCloud, Jira). However, these tools do not support the connection between low-level metrics and assessment models. Very few works have developed specific software artifacts to support such connection, for instance the works proposed by Wagner et al. [21]  and Staron et al. [27, 31, 32]. Most of these proposals do not further detail nor share their developed software artifacts. A relevant exception is  [21] , that provides reusable software artifacts to enable the software quality indicator monitoring. To get insights about the state of the practice in this area, we performed a survey in the context of the industrial partners of the Q-Rapids project. See further information about such survey in  [52]. The main observations were: • None of the organizations used any method or approach to assist them with the specification of software-related indicators to support their decision-making processes.
SESSI: A Method to Specify and Estimate SSIs in Software Development In this section we detail the SESSI (Specification and Estimation of Software Strategic Indicators) method. As mentioned before, SESSI was devised from an action-research like cycle in the context of the industrial partners of the Q-Rapids project. SESSI is aimed to support software-development intensive organizations to specify SSIs based on their available data and to construct SSIs estimation models for supporting decision-making with SSIs monitoring and what-if analysis. SESSI comprises two main phases involving one or more domain experts (i. e. , people from the organization who are familiar with their data sources and their decision-making processes). The first phase provides support to specify an SSI and its associated data collectors (i.
e. , software artifacts allowing automatic data collection from the data sources of the organization). The second phase aims to build the SSI estimation model using a combination of data collected through data collectors and expert knowledge. The resulting model provides SSI estimations, thus enabling SSI monitoring and supporting decision-making. As mentioned in section 2, the SESSI method was conceived from dealing with the industrial needs of the Q-Rapids industrial partners and devising solution attempts in their contexts. Preliminary formative results of the specification and estimation model construction phases are provided in  [16]  and  [17]  respectively. The method presented here extends these preliminary results by consolidating both phases and providing not just results in the context of a single partner but consolidating the experiences from all of them, broadening the applicability of the method.
Phase 1: SSI Specification The specification of an SSI reconciles the informational needs for decision making of the interested organization and the software-related data available in its repositories (e. g. , continuous inspection tools, continuous integration tools, project management tools and issue trackers). To enable the automatic collection of these data as input to estimate the SSI, we suggest the development of data collectors. To determine the information required for the specification of an SSI, several elicitation activities can be performed by the domain experts of the organization. Although SESSI does not propose specific techniques for this purpose, some of the techniques we have used in previous industrial cases may serve as a useful reference  [16]. For illustrative purposes, in the remaining section, we will use as a reference example the Product Quality SSI from one of the Q-Rapids industrial partners.
They are aggregated from factors. For example, an organization may define its Product Quality SSI (see Figure  3 ) as a key aspect of a software product to be considered for decision making. • Data collectors refer to software artifacts that should be developed to automatically collect data from organizational repositories to compute the metrics. Each organization can develop its own data collectors, reuse or customize them from other projects. Some open source data collectors for tools like Jira, OpenProject or SonarQube were developed by the partners of the Q-Rapids project as modular software artifacts integrated in a Java tool available in  [61].
Phase 2: SSI Estimation Model Building The purpose of this phase is to build an estimation model for the SSI specified in the previous phase. It is done by exploiting expert knowledge and historical data gathered through data collectors. The estimation model is built based on Bayesian networks. We chose Bayesian networks as they are a flexible and well-known instrument highly used in software engineering to build estimation models under uncertainty conditions  [34, 35, 54]. The flexible nature of Bayesian networks and the extensive theory and software tools available for their construction and management were important factors that we considered positive to contribute to the industrial uptake of the SESSI method. The availability of several software tools (some of them as open source software) such as Netica®  [64]  or unBBayes  [65]  have allowed us the creation and management of the estimation models through graphical interfaces.
-Estimation Model Generation Once the CPT for every node has been specified, the Bayesian network estimation model can be built using the DAG and CPTs from steps 2 and 3. To build the Bayesian network, in the context of this work, we used Netica®  [64]  and unBBayes  [65]. An example of the resulting Bayesian network for the Product Quality SSI is shown in Figure  7. The figure shows the DAG structure and the probabilities of the nodes, computed from their individual CPTs and the CPTs of their parent nodes (if any). We have attached the two complete CPTs for % Passed Integration Tests and % of tasks of type "Bug" metric nodes (which coincide with their probabilities as they are root nodes), and the partial CPT for the Software Stability factor node. [34, 56]. The SESSI method also applies these validation methods, but in contrast to the mentioned works, SESSI conducts the validation for all the nodes of the model, except for the metric nodes (as their CPTs are directly quantified from the training data and refined by the domain experts).
It is done to compare the resulting state perceived by the domain experts to the most probable state resulting from the estimation model output, after introducing the scenario. If there is a mismatch between these states, the estimation model should be recalibrated by tuning the corresponding CPT and proceeding with the validation until mismatches are amended. The number of hypothetical scenarios to design in this validation can influence the accuracy of the model. However, the recommended number greatly depends on the number of parent nodes and their potential states. 2) Outcome Adequacy. The goal of this second validation is to determine the validity of the Bayesian network with real scenarios from the validation dataset. It is done by comparing the judgements provided by the domain experts with the resulting outputs of the estimation model.
Such real scenarios consist in combinations of states for the parent nodes of the node under validation, directly present in the validation dataset. For each scenario, the domain experts provide the expected state for the node under validation, based on what really happened in such scenario. Hence, the main difference with the previous validation stems from the premise that this validation stands on real scenarios instead of hypothetical ones, thus allowing domain experts to reason on the appropriateness of the resulting state provided by them and by the estimation model. The size of the validation dataset will impact on the number of real scenarios that will arise. If this number is too large, it might be needed to select a subset of them to conduct this validation.
Step 6 -Deployment and use of the SSI estimation model Once the Bayesian network model has been built and validated, it can be deployed and connected to the data collectors to provide automatic and periodical SSI estimations. Netica®  [64]  or unBBayes  [65]  APIs can be used to connect the Bayesian network model with their data collectors. It is up to the organization to determine the architectural way to connect the model to the data collectors. One possibility is, for example, having a software orchestrator in the middle that feeds the SSI estimation model with the periodically collected metrics, then providing the output estimations through a dashboard. Figure  8  shows an example of the Product Quality SSI estimation model connected to the data collectors, which automatically compute the metrics.
Case Study Setting Softeam Group 6  is a software-development intensive organization with more than 1300 employees, providing "high-quality services and solutions in strategy, consulting, finance, digital, big data, analytics, performance and operations". As participant in the Q-Rapids project, Softeam promoted the participation of their subsidiary firms in this study. Modeliosoft 7  , one of its subsidiary development firms, accepted to participate in the study. Modeliosoft representatives opportunistically selected ModelioNG, a software product currently developed and maintained by them as the unit of analysis for applying the SESSI method. Their principal interest was to enable long-term monitoring for the product. The selection of ModelioNG was mainly based on the chance of having a project in an initial stage so that the specification of an SSI and the data collection during the development process would be less disruptive and feasible for the organization.
To understand ModelioNG setting, we conducted semi-structured interviews to gain insights on the roles, needs, processes, information flows and software development related tools used by ModelioNG. The software development process in ModelioNG was defined as "close to agile", focused on the development of working software, face-to-face communication and close collaboration with customers. The overall way of working was characterized as follows: upon analysing the market, Modeliosoft team updates an annual roadmap and elaborates strategies to ensure the success of their long-lived product in a competitive environment. The planned features and requests from customers are then developed, and major issues (if any) are addressed. The code is periodically delivered to integration for nightly builds.
Modeliosoft agreed and assigned the team leader of ModelioNG as the promotor of the application of the method. He provided us access to the software development related data and led the development of data collectors. In addition to the team leader, other roles participated as domain experts: the product manager, the project manager, the developer leader and a developer. The team leader assigned the roles to each step of the method, according to their expertise and availability. Our interaction with these domain experts was direct or indirect, depending on their availability.
Phase 2: SSI Estimation Model Building To execute this second phase of the method, we had to make sure of having enough collected data into the HistoricalNG database. ModelioNG team together with our research team decided to perform the second phase of the method to build the estimation model for the Product Readiness SSI once we had a period of 3 months of collected data into the HistoricalNG database. The domain experts that participated in this phase were: the product manager, the project manager, the developer leader, the team leader and a developer. The steps were conducted as follows: Step 1 -Data Splitting The historical data from the HistoricalNG database was split into 80%-20% for training set and validation set, respectively. The training set was used for the estimation model building (steps 2 and 3) while the validation set was used for validation purposes in step 5.
Step 3 -CPTs Specification To specify the probability function for each node, each CPT was semiautomatically filled in using the training set and the domain experts' knowledge. Three members of the domain experts team participated in this step: the team leader, the project manager and the developer leader. CPTs for metric nodes were automatically computed from the training set, the specified states and their corresponding binning intervals. To do so, we applied frequency quantification with our developed software tool getFrequencyQuantification  [70]. The domain experts reviewed and refined the resulting CPTs according to their knowledge and rules. For instance, the Build Stability metric node (collected from Jenkins as the percentage of successful builds in a specified time period) had 5 ordinal categories defined by the domain experts, ranging from "VeryLow" to "VeryHigh", and binning intervals in the [0, 1] interval.
Outcome Adequacy It aims to validate the model using real scenarios from the validation dataset. The project manager and the team leader participated in this validation as domain experts. Each real scenario from the validation set consisted of the combination of the states of the parent nodes together with the date when the scenario happened. We showed such information to the domain experts and requested them to specify the resulting state for each scenario. Their answer was then compared with the output of the estimation model for that scenario. For instance, for the Product Stability node, there were 10 real scenarios in the validation set, and domain experts provided a potential state for each of these scenarios. 3 out of these 10 scenarios resulted in mismatches between the domain experts' perception and the estimation model output.
Step 6 -Deployment and use of the Product Readiness SSI estimation model After obtaining the final estimation model for the Product Readiness SSI, we discussed with Modeliosoft the most feasible alternatives for its deployment in the organization. Since the very beginning of the case study, Modeliosoft stated their interest on integrating the resulting estimation model into a dashboard  [81]  that was being promoted by its headquarter Softeam to visualize and monitor SSIs. We built the infrastructure to connect the Product readiness estimation model with such dashboard. It was feasible as the dashboard has a software orchestrator component that allows the connection of data collectors with the mechanisms to process the data to be visualized. We developed a software library  [73]  that obtains the collected metrics and returns the resulting set of probabilities for each state of the SSI, which can then be visualized in the dashboard.
This architecture is graphically represented in Figure  11 (b). Modeliosoft was interested on what-if analysis to assess scenarios that could help them to take preventive actions, with the aim of reducing the risk of delivering the software product without meeting its requirements. To enable such analysis we used Netica® software  [64]. Figure  12  shows an example of a what-if analysis conducted with the Product Readiness estimation model. We manually entered a scenario where the development of features is almost finished (that is, Development Task Completion and Specification Task Completion metrics are in "High" and "VeryHigh" states, respectively) but the percentage of minor bugs addressed is low, as well as every metric belonging to the Product Stability factor. In this scenario, the estimation model results in "NotReady" as the most probable state for the Product Readiness SSI.
Conclusions and future work In this paper, we presented the SESSI method, aimed to support softwaredevelopment intensive organizations with guidance and tools for exploiting software development related data and expert knowledge to improve their decision-making. We also illustrated its application in a case study related to the development of ModelioNG, a software product from Modeliosoft, a software development firm. In general, the case study showed that the application of the SESSI method for specifying and monitoring Product Readiness SSI in ModelioNG was smooth and feasible. The obtained estimation model and its associated data collectors enabled monitoring and what-if analysis. In addition, we succeeded in putting forward the infrastructure for the connection of the estimation model with an organizational dashboard.
Acknowledgments This work was supported by the  European Union 's  Horizon 2020 research and innovation program  ( Q-Rapids  project, grant agreement  N° 732253 ), and partially by the  Spanish Ministry of Economy and Competitiveness  (project  GENESIS , grant agreement  TIN2016-79269-R ). We also thank  Softeam and Modeliosoft  for promoting and participating in the case study, respectively.