dataesr/scientific-paragraphs-categorization huggingface dataset:
Repairnator and Repairnator-Experiments Repairnator is a software engineering bot that monitors program bugs discovered during Continuous Integration, and tries to fix them automatically  [36]. Repairnator implements multiple repair strategies, including the generation of code-removal patches using jKali. Repairnator-Experiments is an open science repository 4  that contains the metadata information of the Travis CI builds that Repairnator tried to repair  [49]. It hosts 14,132 failed builds (collected in the period February 2017 -September 2018) for 1,609 Java open-source projects hosted on GitHub. It provides detailed information about the builds, such as the event that triggered the build, the number of failing and crashing test cases, and the type of failures. Moreover, for every build, Repairnator-Experiments stores the source code associated with the failing commit.
Definition of Failing and Crashing Test Cases The projects in the Repairnator-Experiments repository are equipped with automated unit test cases. In our study we used these tests to validate the patches generated by jKali. When a test fails, we distinguish two main cases: failing test cases and crashing test cases. Definition: Failing test case. A failing test case is a test case that fails due to a violated assertion. Typically, it means that the actual values generated by the program under test violate one of the assertions present in the tests. Per the terminology of JUnit, this is a test failure  [19]. Thus, a failing test case reports an invalid test result. Definition: Crashing test case. A crashing test case is a test case for which the program under test generates an exception during its execution.
Goals & Research Questions The goal of this study is to understand the nature of test-suite-adequate coderemoval patches and to investigate how these patches relate to human patches, with a thorough qualitative study. To our knowledge, this is novel in program repair research, nobody has ever studied this important point. In particular, the study aims to answer to the following research questions: RQ1 What is the relation between assertion failures and the gener- The goal of this research question is to study if developers fix bugs according to some patterns when the bug can be also addressed with a code-removal patch. We also investigate the semantic and syntactic similarities between the fix produced by developers and the patch produced by jKali. Our experimental methodology is organized in three main phases: the collection of build failures, presented in Section 3.
Data Collection The first phase of our study consists of collecting the build failures, and the related artifacts, necessary to answer to RQs 1-4. As source of build failures, we consider the Repairnator-Experiments repository, which includes thousands of past failed builds annotated with meta-data as described in Section 2. 3. For each failed build, we consider several artifacts: -The failure-related artifacts. -The code-removal patches generated for the failed builds. -The human patches associated with the failed builds that also have a coderemoval patch. All these data have been saved on a GitHub repository 7  for the sake of scientific reproducibility. We describe below how we obtain these artifacts. Failure-related artifacts. The builds relevant to our study are all the builds stored in the Repairnator-Experiments repository with either one failing or one crashing test case.
We found 2,381 builds with only one failing test case, and 1,724 builds with only one crashing test case. Overall 4,105 out of the 14,132 builds (29. 05%) satisfy our selection criterion. For each failed build, we collect from the Repairnator-Experiments repository the source code version that made the build fail, the metadata associated with the failed build (e. g. , the build id), and finally the information about the failure (e. g. , the name of the class and the test method that fails or crashes). Code-removal Patches. For every selected build, we ran jKali to generate code-removal patches, if such patches exist. We did not use the coderemoval patches already present in the Repairnator-Experiments repository because they have been generated with a previous version of jKali, which was affected by several bugs now fixed.
These new runs ensured that a project code associated with a failed build is still executable. In fact, sometime there are problems with deletion of branches or dependencies that now make the execution of jKali impossible. We set jKali to find every possible patch in the search space, not stopping after finding the first patch. The search space ordering used by jKali is given by the suspiciousness score of the faulty lines computed with fault localization, implemented in GZoltar  [5]. GZoltar returns a list of lines associated with a suspiciousness score, ranked from the most suspicious to the least suspicious one, according to the Ochiai formula. jKali, starting from the most suspicious line, tries to generate a code-removal patch by changing the program in the considered line. The code-removal patch can be generated using any of the following three operators as follows: 1) remove a code element, 2) replace a condition in order to force the execution of a specific branch, and 3) add a return statement to interrupt the execution of a procedure.
Thus, for every failed build, there could be 0, 1 or multiple code-removal patches. Our experiment was performed using a CPU Intel Xeon E5-2690v4 (2x14 cores) with 128GB of Memory per node. All nodes are running Ubuntu Focal (20. 04 LTS). jKali was run in parallel on different builds thanks to the PFS file system available on Swedish National Infrastructure for Computing (SNIC). Given the computational nature of the task, we run jKali with a timeout of 3 hours for every build. This 3 hours period comprises all phases of jKali, from downloading the source code from the repository to the execution of the repair attempts. The core repair loop of jKali itself is set to run for a maximum of 100 minutes. This setup is based on previous research  [11] , in which repair tools required 13. 5 minutes on average to generate a patch.
Thus, allocating 100 minutes is 7. 4 times the average repair time, and it reduces the probability of stopping the repair process due to a timeout. Using this configuration, in our experiments jKali never reached the timeout. This process produces in total 129 code-removal patches on the considered bug benchmark (65 patches for the 28 builds with a failing test case and 64 patches for the 24 builds with a crashing test case). Human Patches. The data collection step finally includes the identification of the human patches associated with the builds for which jKali created a code-removal patch. To determine the corresponding human patch, we combine automatic and manual analysis: we use the Travis CI API 8  to automatically determine the commits that may include the patch and also we manually check the presence and appropriateness of the human patch.
Environment Misconfiguration The failure is caused by an incorrect execution environment (e. g. , an environment variable is not set). ous studies  [30, 39] , we consider a patch to be correct if it is either identical or semantically-equivalent to the corresponding human patch. In Listing 2, we exemplify the case of a code-removal patch generated by jKali for the failing Travis CI build with id 322406277 12  associated with the project pac4j. This patch is semantically-equivalent to the human patch, and is thus considered correct. The code-removal patch changes the if statement using false as condition, which forces the execution of the else branch, and consequently forces the method to always return null. The human patch removes the overridden method shown in Listing 2  13. The program without the overridden method has the same behavior as the program with the coderemoval patch.
This instruction is executed only if the JSON object created at line 3 has the property "priority". The code-removal patch generated for this build and shown in Listing 8 removes the instruction to add the property "priority" to the JSON object, and in this way the program passes the test case. @org. junit. Test public void s e r i a l i z e T e s t() { J S O N O b j e c t ds = todo. s e r i a l i z e() ; a s s e r t E q u a l s( todo. getTitle () , ( String ) ds. get ( " title " ) ) ;. if ( ds. has ( " priority " ) ) a s s e r t E q u a l s( todo. g e t P r i o r i t y(). toString () , Finally, the presence of flaky tests  [28]  may let jKali accept a patch. When a flaky test stops failing, it has actually no causal relation with the generated patch, and the patch is then wrongly assumed as being correct.
Not Found It is not possible to determine the changes that make the patch, for instance because they are mixed with many other changes not related to the removed fault. The first two categories, Fix Test Code and Fix Test Data, correspond to patches implemented by changing the code of the test, while distinguishing between changes to the logic of the tests and changes to the data used in the tests. A number of categories capture the case of actual modifications implemented in the code of the faulty program (excluding code-removal only patches). These changes might be at the level of the individual statements or at the level of the methods. Changes to individual statements involved either conditions or if-else statements. Although other types of changes to individual statements are possible, we do not list patterns for which no code-removal patch exists in our dataset.
Analysis of the Results The most frequent category of assertion failure is Wrong Values with 842 occurrences, while the least frequent category is Environment Misconfiguration with 2 occurrences. jKali generates a patch for all failure categories, with the exception of Environment Misconfiguration. This is due to both the few occurrences in this category, but also to the missing capability of changing the environment configuration files in jKali. In fact, jKali is designed to create patches that change the source code of the target program, and cannot make any change to the environment. Wrong Values is the category of assertion failure with the highest number of occurrences (842) and the highest number of code-removal patches  (22). The high number of patches is only due to the high number of failing builds in that category.
Comparison of the Results with Previous Studies Another interesting aspect is that the proportion of generated code-removal patches is significantly lower than in previous studies. Indeed, in our case, the likelihood is 2. 95%, while in the previous studies is 36. 23% [27] , 25. 71%  [39] , and 9. 28%  [30]. Our best explanation is that it is due to the small size of the datasets used in former experiments, which consist of 69 cases for  [27] , 105 cases for  [39] , and 224 cases for  [30]. A second explanation is that those previous studies did not sample over builds but over commits. A third explanation is that the benchmarks of those studies used some kind of selection, which results in a biased sampling. Yet, our result is aligned with Durieux et al. [11] , in which the proportion of patches generated by jKali on the bugs of Bears is about 3.
0%. Unlike the other benchmarks, Bears is a benchmark which uses CI builds to identify buggy and patched program version candidate, and it contains bugs associated with more different programs (72 projects) compared to the other ones (8 for ManyBugs benchmark  [20] , and 5 for Defects4J benchmark  [16] ). Both Bears and this study sample builds, which explains the strong consistency. What is the relation between assertion failures and the generation of test-suite-adequate code-removal patches? (RQ1) jKali has been able to create a patch for 28 out of 949 (2. 95%) builds having only one failing test case. We obtained one patch for every category of assertion failure with the exception of Environment Misconfiguration, which is out of scope for current generators of code-removal patches. Our results show that the generation of a code-removal patch is independent of the category of assertion failure.
We explain this result by the fact that these exception types are prevailing in the benchmark. Furthermore, we manually analyzed them. We observe that jKali successfully patches programs producing a NullPointerException by removing the usage of the object that is null. Regarding the generic exception java. lang. Exception, the 2 coderemoval patches are related to timeout errors and they remove the piece of code that causes the timeout. For example, the code-removal patch of the Travis CI build for Apache Twill (id 356030973) removes the call to method java. net. InetAddress. getLoopbackAddress(), whose execution could generate a deadlock. Some exceptions only sporadically present in the benchmark have been successfully patched, that is the case for MQClientException  16  , ConnectorStart-FailedException  17  , and PersistenceException 18.
Buggy Test Case For 9 out of 52 builds (17. 31%), the code-removal patch reveals a Buggy Test Case, that is, a faulty test case that allows the acceptance of an incorrect patch. For example, build 35121194 of our dataset fails after the addition of a change that trims the output associated with the result of a command execution, but the test case is not updated to support this change. The coderemoval patch works because it removes exactly the new instruction that trims the output (i. e. , the patch undoes the change), and so the test case passes. To our knowledge, this is the first ever report of this phenomenon in the literature.
In particular, for the build 384760371, the human fix and the code-removal patch are disjoint because they change different points of the source code, but they are semantically related because both changes influence the same value used by the program to save records in a database. Interestingly, considering both the 24 code-removals patches that work due to weak test suites and the 9 ones that work due to buggy test cases (33 builds in total), in 18 out of 33 cases (54. 55%) the human fix precisely consists in fixing the test code (13 cases) or the test data (5 cases). This finding confirms the relation between the quality of the test suites and the ability of jKali to generate code-removal patches. Flaky Test When a code-removal patch works because of a flaky test, in 7 out of 8 cases (87.
Analysis of Builds with More Than One Code-removal Patch Finally, we now consider the cases for which jKali is able to create more than one code-removal patch. Considering the builds with one failing test case, it was possible to generate more than one code-removal patch for only 9 out of 949 (0. 95%). Considering the builds with one crashing test case, jKali created more than one code-removal patch for only 7 out of the 969 (0. 72%) builds. We now analyze these cases in details. Incorrect Patches Some patches are simply incorrect. For instance, the second patch (ID 2) generated for build 384713759 is incorrect since it prevents the program from throwing the javax. persistence. PersistenceException, by removing the instruction that stores the data in the database. Thus, the crashing test case is turned into a passing test case by removing a functionality.
Weak or Incorrect Test Cases Some additional code-removal patches are incorrect because of problems in the test suite. For example, considering build 400611810, jKali manages to create 6 code-removal patches, all incorrect. This suggests there is a problem in the tests. Indeed, all the patches, in different ways, avoid to set the header value. In this way, the request has the correct status code expected by the test case, but the header is empty, making the program incorrect, although it passes the test cases since no test checks the content of the header. Another example is build 408694507 for which jKali creates 3 code-removal patches. These patches, in different ways, avoid the execution of a return statement introduced by the developer in a recent change. The developer did not update the test case to support the new behavior of the program, and for this reason a test failed.
Both code-removal patches are related to the same part of the program with the difference that patch with ID 1 deletes only a subset of the piece of code removed by the patch with ID 2. This means that the developer could focus only on the logic of the code removed by Patch 1, since the extra piece of code deleted only by Patch 2 does not change the final result. Flaky Test It happens that code-removal patches change the program in an opposite way. Indeed, in 3 out 4 cases (builds 374587117, 407166687, and 415796275) jKali creates this type of patches. An example is reported in Listing 12 and Listing 13, that show two code-removal patches generated by jKali for build 407166687. In the first case, Patch 8, the if condition is changed with the keyword true forcing the execution of the then-branch, while in the second case, Patch 9, the if condition is changed with the keyword false, forcing the execution of the else-branch.
The studies by Long and Rinard  [27]  and Qi et al. [39]  consider code-removal patches in C programs. Both studies use the ManyBugs benchmark  [20] , which is the one used to evaluate the original implementation of GenProg, but Long and Rinard restrict their investigation to 69 bugs, since they classified the rest of the bugs as not being actual detects. Their results show that most of the code-removal patches exist because of a weak test suite. In the experiment conducted by Martinez et al. [30] , there are 22 coderemoval patches, but only one is correct. They also report that the test cases influence the results about the generation and correctness of a patch. Indeed, if the test cases insufficiently cover the program functionality, program repair tools can generate patches that make the program pass all the test cases, but they can also introduce some bugs not revealed by the test suite.
Analysis of Patches A primary concern about program repair techniques is the quality of the generated patches. Indeed, several studies show that plausible patches are often unsatisfactory for developers. For instance, the study by  Qi et al. shows that the majority of the patches generated by tools like GenProg simply delete functionality, resulting in incorrect code changes  [39]. This paper extensively studies code-removal patches, investigating their causes, and the information that can be extracted from the patches, even if incorrect. The study by Martinez et al. [30]  based on the empirical comparison of three repair techniques (jGenProg  [32] , jKali  [32] , and Nopol  [52] ) points out the problem of overfitting patches, which are test-suite-adequate patches that fail to repair the target bugs.
Similarly, the study by Ye et al. [54]  investigates the effectiveness of program repair tools on the QuixBugs  [22]  benchmark, confirming the challenge of overfitting patches as one of the main challenges for program repair techniques. In particular, the study shows that jKali is able to generate three patches for two different programs, and only one of them is correct. This result is aligned with our findings, where only 2 out of the 52 (3. 85%) generated patches are correct. One of the biggest experiment that considers 2,141 bugs belonging to 5 different benchmarks and 11 repair tools  [11]  analyzes the effectiveness of program repair techniques identifying factors that prevent the generation of patches. In particular, the study shows that jKali is able to fix 52 out of 2,141 bugs (2.
For this reason, we focused our study on code-removal patches on cases with a single failing or crashing test case. 6. 4 Code-removal in Other Patch Generation Techniques Some program repair techniques although not exclusively focusing on coderemoval patches, as Kali does, can be able to generate them. We now discuss the presence of code-removal patches in other program repair techniques. Ye et al. [55]  collected patches for the bugs in Defects4J benchmark generated by 14 different repair techniques. In this dataset there is only one bug, Math-50, for which different repair tools (e. g. , jGenProg  [32]  and ELIXIR  [42] ) generate a correct code-removal patch, and there are 12 bugs for which a plausible code-removal patch has been generated. In a recent study on another benchmark, QuixBugs  [54] , the authors analyzed the effectiveness of the repair strategies implemented in different tools.
According to the reported data, jGenProg generates 8 code-removal patches out of 164 patches (4. 88%), while jKali creates 3 code-removal patches. Note that 6 out of 8 code-removal patches generated by jGenProg are for the same bug. There are also other techniques that leverage templates extracted from human-written fixes in order to automatically create new patches. Liu et al. analyzed the recurrently-used fix patterns in automatic program repair tools, and they reported that 4 out of the 35 identified patterns implement delete actions  [25]. In particular, among them, only 2 out of 4 can be exploited to create code-removal patches as jKali does, because the other two patterns remove arguments from method invocations if the method has overloaded methods and remove some of the expressions in conditional statements.
They also implemented the repair patterns in a tool called TBar, and they tested them on the Defects4J benchmark. Based on the data reported in the repository associated with the experiment, 20 bugs have been fixed with a code-removal patch. SOFix is another template-based technique in which the repair templates have been extracted from the answers written by users in the Stack Overflow forum 23. In particular, they mined 13 repair templates, and among them 2 are for generating code-removal patches. In their evaluation on the Defects4J benchmark, SOFix fixes 2 bugs with a code-removal patch. As TBar and SOFix, also jKali has been evaluated on Defects4J benchmark, and it fixed 27 bugs  [11]. Thus, based on these data, although different techniques may generate code-removal patches, they actually implement additions and code replacements most of the time.
Thus, deleting code is not useful only to fix and improve the codebase of a program as jKali and Piranha do respectively, but it can be used also to better manage the external dependencies, removing the ones that are not necessary. Humans tend to add features instead of removing them  [34] , and we believe developers are the same when it comes to software engineering. Yet, deleting code is clearly an option to manage code ranging from program repair to software debloating.
Empirical Studies Investigating how Humans Use Automatic Patches Finally, there are studies to understand how humans work with the patches generated automatically by program repair techniques. Indeed, program repair techniques may produce patches that are not accepted by developers  [48]. The early study by Fry et al. [13]  measures the maintainability of patches, and it shows that machine-generated patches are slightly less maintainable than the human-written ones. Tyler et al. [41]  show that developers in general tend to trust more in patches produced by humans than automatic program repair techniques. Similar results are the ones obtained by Alarcon et al. [1] , indeed, they report that programmers find human repairs more trustworthy than the ones generated by GenProg. The study by Cambronero et al.