dataesr/scientific-paragraphs-categorization huggingface dataset:
In most active object languages, future references can be transmitted between remote entities without requiring the future to be resolved. There are different models related to active objects. The ASP calculus  [3]  is a distributed active object calculus with futures; the ProActive library is its reference implementation. ASP has strong properties and is easy to program, but the strict sequential service of requests creates deadlocks. A different approach is provided by active objects languages with cooperative multithreading (Creol  [7, 10] , Jcobox  [8] ). In this model a thread serving a request can stop in order to let the service of another request progress. While no data race condition is possible, interleaving of the different request services triggered by the different release points makes the behaviour more difficult to predict; in particular Creol does not feature the determinism properties of ASP.
Comparison with Related Work Parallel actor monitors  [16]  (PAM) provide multi-threading capacities to actors based on explicit scheduling functions. However, we believe that the compatibility annotations of multi-active objects provide a higher level of abstraction than PAM, and that this high-level of abstraction is what makes active-objects and actors easy to program. The main difference between our approach and active-objects with cooperative multi-threading like JCoBox  [8]  and Creol  [7]  is twofold. On one hand, Creol-like languages are not really multi-threaded (only one thread is active at a time  [7] ), thus they do not necessarily address the issue of efficiency on multicore architectures; JCoBox proposes a shared immutable state that can be used efficiently on multicore architectures but as the distributed implementation is still a prototype, it is difficult to study how an application mixing local concurrency and distribution like our CAN example would behave.
On the other hand, concerning synchronisation, in cooperative multi-threaded solutions between explicit release points (awaits) the programs are executed sequentially. Adequately placing those release points is the main challenge in programming in Creol or JCoBox: too many release points leads to a complex interleaving between sequential code portions, whereas not enough of them will probably lead to a deadlock. Multi-active objects provide an alternative approach by allowing local concurrency in active objects: with annotations, the programmer can reason on high-level compatibility rules, and parallelism can be expressed in a simple manner. Compatible methods are run concurrently, with potential race-conditions but also local multi-threading. Overall, in ASP and MultiASP distribution and concurrency are much more transparent than in JCoBox and Creol; this difference in point of view explains most of the differences between the two models: transparent vs.
The originality of our contribution lies in the interplay between the formal and precise study of the MultiASP language, and a middleware implementation efficient enough to compete with classical multi-threading benchmarks. The use of dynamic constraints for compatibility allows a fine-grain control over local concurrency and improves expressiveness. We implemented the proposed model in Java and ran experiments to ensure that our approach is efficient. The experiments showed that the performance of multi-active objects is similar to the manually multi-threaded version while code dedicated to parallelism is much simpler when using multi-active objects. We also illustrated the performance gain brought by multi-active objects compared to a classical active object version. Overall, multi-active objects outperform simple active objects, and are easier to program than classical multi-threading.
Empirical Evaluation of Bug Linking Tegawendé F. Bissyandé 1 , Ferdian Thung 2 , Shaowei Wang 2 , David Lo 2 , Lingxiao Jiang 2 and Laurent Réveillère 1  1 LaBRI, University of Bordeaux, France 2 Singapore Management University, Singapore {bissyand,reveille}@labri. fr,{ferdianthung,shaoweiwang. 2010,davidlo,lxjiang}@smu. edu. sg Abstract-To collect software bugs found by users, development teams often setup bug trackers using systems such as Bugzilla. Developers would then fix some of the bugs and commit corresponding code changes into version control systems such as svn or git. Unfortunately, the links between bug reports and code changes are missing for many software projects as the bug tracking and version control systems are often maintained separately. Yet, linking bug reports to fix commits is important as it could shed light into the nature of bug fixing processes and expose patterns in software management.
Bug linking solutions, such as ReLink, have been proposed. The demonstration of their effectiveness however faces a number of issues, including a reliability issue with their ground truth datasets as well as the extent of their measurements. We propose in this study a benchmark for evaluating bug linking solutions. This benchmark includes a dataset of about 12,000 bug links from 10 programs. These true links between bug reports and their fixes have been provided during bug fixing processes. We designed a number of research questions, to assess both quantitatively and qualitatively the effectiveness of a bug linking tool. Finally, we apply this benchmark on ReLink to report the strengths and limitations of this bug linking tool.
Links between bug reports and bug fixes are therefore not readily available to researchers or practitioners to analyze. To address the problem of bug linking, a number of solutions have been proposed. Most of the solutions that aim to establish bug links rely on the fact that meticulous developers, when pushing a fix into the code version control system, always insert specific information that identifies the corresponding bug  [9]. Thus, these solutions can establish bug links based on heuristics to match a set of indicative keywords (e. g. , Fixed, Bug) and the corresponding bug identifiers (e. g. , #1234) in code change logs with those in bug reports  [9] -  [11]. Sureka et al. have used a probalistic approach to trace such links  [12]  Other research work has shown that available datasets in both bug tracking and version control systems are actually plagued by quality issues and require bug linking solutions to be augmented with heuristics for verifying the correctness of their results  [13].
The Linkster tool was designed in this respect to enable an expert developer to quickly find, examine, and annotate relevant changes that were identified through heuristics  [14]. However it does not solve the problem of incompleteness and bias in datasets as many "missing" links cannot be uncovered with these heuristics. ReLink extends previous bug linking approaches by implementing an information retrieval based solution  [15]. Using similarity metrics, ReLink is able to find up to twice more links found by previous approaches. To evaluate ReLink, however, the authors of ReLink used as the ground truth a dataset with links that were manually labeled by themselves and a posteriori by an Apache Web Server developer for their Apache Web Server dataset. Several issues are then raised by this process: 1) The collected "ground truth" is quantitatively constrained by the tediousness of a posteriori manual labeling.
2) The data may be plagued by bias as the labelers are not the actual bug fixers, thos who without doubt could link a bug report with all, and only, the commits that address it. Furthermore, the effectiveness of ReLink has been evaluated only against traditional approaches without introducing variations in the input data, such as the quality and quantity of training data used in their bug linking process. We undertake to build a benchmark for evaluating the effectiveness of bug linking tools with a dataset of 10 programs 1 , and we provide a more extensive evaluation of ReLink. To build the benchmark dataset, we investigate a set of clean data where the links between bug reports and code revisions that fix the bugs are well maintained. We perform a 1 ReLink was originally assessed on 3 programs 1 manual check to verify that links in the dataset are sound and complete.
These links are inserted by the actual developers during their actual bug-fixing activities over a long period of time. The benefit of using such a dataset is clear: we can get a large number of highly accurate bug links, and with this ground truth, we can propose various dimensions to test the effectiveness of bug linking tools. In addition, there are many other widely-used information retrieval techniques besides the one used in ReLink, such as Vector Space Modelling (VSM), Latent Semantic Indexing (LSI), and Latent Dirichlet Allocation (LDA) (c. f. [16] ); they have been shown to be effective for many software engineering tasks, such as software traceability  [17] -  [20]. Thus in this study, we adapt these existing IR techniques for bug linking and compare them against ReLink. These experiments are performed to gain insights on the effectiveness of the IR technique that was used in the bug linking tool.
We find that the ReLink tool offers good precision in recovering links that are actually correct, but is less effective in recovering missing links. We also found that training data has limited impact on the results of ReLink due to other filtering steps used by ReLink to consolidate its outputs. Cross-project validation has shown that, overall, training data cannot be borrowed across different projects. Finally, our comparison between ReLink and other standard Information Retrieval solutions shows that there is still room for improving the effectiveness of ReLink. The contributions of this work are as follows: 1) We provide a benchmark dataset 2 of known true links that could be used to evaluate bug linking work, and discuss a number of research questions for assessing the effectiveness of a bug linking tool.
2) We evaluate the effectiveness of ReLink, a recently proposed bug linking tool, on the benchmark. 3) We compare the effectiveness of ReLink versus standard information retrieval approaches that have been used in prior studies on software traceability. 4) We qualitatively characterize the kinds of links that ReLink misses and others that ReLink wrongly assigns. The structure of this paper is as follows. In Section II, we describe bug linking and ReLink in more details. Section III describes the dataset that we use as the benchmark. We elaborate how we obtain and use this dataset. Section IV details the research questions and the metrics that we use for assessing a bug linking tool. We describe our evaluation results in Section V. We provide a list of related studies in Section VI. We conclude with future work in Section VII.
To select features of links, the ReLink authors have first performed a manual analysis of some explicit links. We briefly describe in the following the features considered in ReLink. Time Interval: The ReLink algorithm considers the interval between the bug-fixing time and the change-commit time to filter out false positives and confirm the possibility of a link using a threshold inferred from the explicit links that were identified by previous heuristics. Bug owner and change committer: ReLink authors have performed an empirical study of explicit links to establish that, often, there exist some relationships between change committers in software repositories and bug owners in bug tracking systems. Indeed, they have observed that although the person committing the bug fix is not always the one responsible for handling the bug report, the mapping between them could be identified, e.
The ReLink paper for automatic recovery of missing links reported very good performance: the average precision rate was 89% and the average recall was 78% for a limited dataset containing 3 projects. However, the ReLink authors relied on a manually labeled "ground truth" for their experiments. Given the importance of bug linking, we believe that it is necessary to more thoroughly evaluate ReLink and assess its effectiveness on a variety of projects, a variety of training datasets, and a variety of usage scenarios to effectively establish its strengths and limits with regards to bug linking.
III. BENCHMARK DATASET Our bug linking evaluation is performed based on a benchmark dataset collected from ten open source projects hosted by the Apache Software Foundation 4. These programs are described in Table  I  with the number of bug reports considered for each program and the number of labeled links collected. Overall, the dataset includes about 7,000 bug reports fixed by one or more commits, thus leading to around 12,000 bug links in the dataset. Our choice of these software systems is influenced by (1) the capabilities of JIRA  5  , a commercial bug/issue tracking system used by Apache projects, (2) the maturity of the projects, (3) the diverse application domains of the projects, and (4) the different programming languages used in the programs. JIRA has various features that make it a desirable toolkit for dealing with bug reports.
For example, as a votingbased system, JIRA is often relied upon for effectively prioritizing important issues (e. g. , the ones that interest users the most)  [23]. Another feature of JIRA is that it provides add-ons for connecting issues to version control systems. In the case of the programs used in our benchmark dataset, the Apache JIRA-based issue tracker  6  was linked to the Apache subversion repository  7  , allowing links to be automatically inferred when commits are checked into the repository. One benefit of the JIRA add-ons is that it is very convenient for programmers to refer to the bug report they are addressing in their commit logs. Thus, there are immediately more opportunities for recovering links with improved quality. In JIRA, issue/bug identifiers are composed of two parts separated by a dash: a keyword that identifies the project (e.
Repository: Revision Date User Message ASF #151795 Mon dnaber increase version number from 1. 5 Feb 07 to 1. 9, so that the jar is called called LUCENE-1. 9-rc1-dev. jar In order to use labeled links as the ground truth we have set to manually assess those links to ensure that they were properly inferred. Soundness: This term refers to whether the labeled links are correct. To assess the soundness of the labeled links, we have randomly sampled 100 links distributed across the ten projects and manually checked whether these were true links. We have found that 100% of those links were true. The link example in Figure  3  actually refers to a revision number that is part of a branch, thus outside the development trunk considered in our benchmark dataset. This empirical evaluation reveals that Apache developers are meticulous in their efforts to insert bug references in the change logs of their fixing commits.
However, we have also noted that some bug reports were automatically imported from a previous Bugzilla-based setup of the project's bug tracking system into the JIRA setup, which may reduce the reliability of links that were not labeled by bug fixers. We have therefore parsed all labeled links from our datasets and found that 37 out of 6507 involved a bug report with an initial Bugzilla ID. Manually checking these links exposed 22 wrong links with LUCENE-3. In all these links the expression "LUCENE-3", which was actually part of the release version number "Lucene3. 1. 0", was mistakenly inferred as a bug number during the automated labeling by JIRA. We have then removed these links from the benchmark dataset. Completeness: This term refers to whether all links between bug reports and code commits are labelled.
Completeness is an important property that should be ensured to accurately evaluate false negative rates of bug linking tools. We find that every issue/bug in our benchmark dataset has been linked to some commits. In addition, for links provided by JIRA for the issues/bugs, we want to make sure whether all code commits related to an issue/bug have been linked to the issue/bug in JIRA. For this purpose, we randomly sampled 100 links to check whether a bug report may be related to a commit not linked by JIRA. Two situations may explain why a given commit is not linked to a bug report. First, the commit may not be a bug-fixing commit, or it may fix a bug that was not reported in the bug tracking system. Second, the commit may be part of a number of commits addressing the same bug. In this case, we assume that the unlinked commit was caused by missing references to its corresponding issue/bug report in its commit log when it actually belongs to the fix split in several commits.
We have investigated the dataset and found that when a bug is fixed by many commits, those are usually close in time, and for the ten programs, the number of commits do not reach 20 for a given bug. Thus, for each linked commit in the sample set, we manually examine the 10 commits that precede it and the 10 commits that follow it to see whether there are unlinked commits for the same issue/bug. Our manual investigation has established that 100% of the sampled links were complete. The results of these investigations allow us to use the links extracted from the JIRA-based issue tracking system with little clean-up as our ground truth for evaluating bug linking tools.
Variations in real-world datasets may therefore impact the performance of such bug linking tools. Consequently, for a bug linking tool that relies on machine learning approaches, it is important to investigate its sensitivity when training data is changed. RQ4. Could the tool be trained on one software system and used to link reports in other software systems?: Related to the previous research question, a worst case scenario may arise when no training data can be found in the project. For example, for the first bug report in a software project, there is no training data available. Because explicit links are not readily available in all real-world projects, a bug linking tool would be more valuable if it can use training datasets from one project to infer links in another. We explore in this research question if the thresholds learned in one software system could be used in other systems.
V. RELINK EVALUATION RESULTS In this section we report the evaluation of ReLink with our benchmark dataset and following the research questions previously outlined. The discussions are based on the metrics described above. For our evaluation process, we use the ReLink tool as a black box, only providing desired inputs and analyzing the outputs. We have downloaded the version of the tool that was available at the project web page 8  at the time of writing. Since our ground truth always contains references to the bug reports that are involved in a link, which would hinder the evaluation of ReLink's capability in finding "missing links", we accordingly pre-process the inputs of change logs to remove the references from the portions of data that are used for testing, but leaving them in the training data for ReLink to infer link features using traditional heuristics.
9 A. RQ1: Link Effectiveness (Non-Linked) Since ReLink relies on a learning algorithm, k-fold cross validation is a well suited statistical method to evaluate its effectiveness  [24]. To answer the first research question we therefore perform a 10-fold cross validation for each of the programs in our benchmark dataset. For this purpose, we have randomly distributed the labeled links into 10 sets of equal size. For each program, we ran 10 experiments using every time 1 set as the testing set and the 9 others for training data. The results are shown in Table  II  for all programs. Discussion: These experiments show that, in general, the ReLink tool has good precisions, reaching 100% for the struts program, though this precision can drop in some cases, as for the stdcxx program. Recalls, however, are very low, which in turn cause low F-measures.
C. RQ3: Sensitivity to Training Data To assess the sensitivity of ReLink to training data, we perform a series of experiments with varying sizes of the training data. Practically, for each program, we have randomly distributed the labeled links into nine buckets of potential training data. We then run 9 successive experiments where in the first experiment labels from the first bucket are used for training, and in the second experiment we add labels from the second bucket to double the size of training data, and so on. This experimental scenario enables us to consider from 10% to 90% of the labeled links as training data and the remaining, i. e. , from 90% to 10% as testing data. Figures  4, 5  and 6  show the results of our experiments for the different programs. Discussion: In Figure  4 , we note that precision is not significantly impacted by the size of training data.
We believe that this is due to the fact that ReLink uses different heuristics aside from the similarities between change logs and bug reports, to consolidate its outputs. The impact on recall rates is observable on datasets of the highest and lowest sizes. Figure  5  shows that recall is increasing for the felix program and overall is dropping for the struts program. These results suggest that smaller datasets, in which outliers are more noticeable, would affect the recall of ReLink. Finally, in Figure  6 , we notice that the F-measure follows the results of the Recall metrics. This was expected based on the results of Precision which did not appear to be affected by the variation in proportion of training data.
D. RQ4: Cross Project Effectiveness To answer the research question related to cross project effectiveness of ReLink, we ran experiments with all combi-   7  and 8. Discussion: From the graphs detailing the precision and recall results, we observe that, overall, using training data from other projects datasets leads to lower precision and recall. In a few cases, such as with the mahout and opennlp programs, smaller sets of training data (e. g. , from struts) have less impact on the precision of ReLink and may even improve it slightly.
E. RQ5: Comparison with Other IR Solutions To answer this research question we use several standard information retrieval techniques namely vector space modeling (VSM), latent semantic analysis (LSA), and latent Dirichlet allocation (LDA), which have also been used in studies on software traceability analysis. Following is the description of how we use these techniques for bug linking. Practically for the purpose of fair comparison, we have implemented the considered standard information retrieval solutions to follow the same steps as described in the ReLink paper. These solutions perform a simple retrieval without considering some features of links (time interval and mapping between bug owner and change committer). The process for each model is the same. First, they preprocess the text data through stemming and stop words removal.
Second, they take bug reports as query and search the relevant change logs for this query. For every bug report, the similarity scores between its text data and change logs are computed based on the model in use (VSM, LSA or LDA). Links are then inferred by selecting change logs for which the similarity score is above a threshold which was determined in a training phase as in ReLink. To compare ReLink and the aforementioned IR techniques, we resort to 10-fold cross validation. The experiment scenario for each technique is similar to the one used for answering the first research question (RQ1) for ReLink. The results of these various IR techniques as compared to ReLink is shown in Table  IV. From the table, we could notice that ReLink outperforms the existing IR techniques in terms of F-measure for: activemq, felix, lucene, mahout, struts, xalan, and xerces.
VSM outperforms ReLink for: hadoop, opennlp, and stdcxx. LDA and LSA outperform Relink for: opennlp and stdcxx. Thus in general, ReLink is better than existing IR approaches. The existing IR approaches are promising too as it could outperform ReLink on 2 or 3 out of the 10 programs. In the future, it would be interesting to propose an approach that could extend ReLink such that it could outperform all existing IR techniques on all datasets.