dataesr/scientific-paragraphs-categorization huggingface dataset:
Curated Patch Dataset Fourteen repair systems. APR patches for Defects4J form the essential data for our experiment. The criteria of repair systems considered in this study are that they were previously evaluated on the Defects4J  (Just et al. , 2014)  benchmark. We carefully collect APR patches that are publicly available. We perform this by browsing the repositories / appendices / replication packages of the corresponding research papers or by asking the authors directly. As a result, we build our dataset D correct and D overf itting from following 14 APR systems: ACS  (Xiong et al. , 2017) ; Arja  (Yuan and Banzhaf, 2018) ; CapGen  (Wen et al. , 2018) ; DeepRepair  (White et al. , 2018) ; Elixir  (Saha et al. , 2017) ; HDRepair  (Le et al. , 2016) ; Jaid  (Chen et al. , 2017) ; JGenProg  (Martinez et al.
New Generated RGT Tests In this paper, we decided to generate new RGT tests for two main reasons. First, we execute 30 runs of Evosuite  (Fraser and Arcuri, 2011)  and Randoop  (Pacheco and Ernst, 2007) , using a different random seed value on each, with the goal of generating new test cases (not generated by the 10 executions from  (Shamshiri et al. , 2015) ) that potentially detect behavioral differences. They are respectively denoted as RGT Evosuite2019 and RGT Randoop2019. By using 20 additional executions with new seeds, the new test cases sample other parts of the input space. Second, the test dataset from Evosuite EM SE18 partially covers the Defects4J bugs (42 in total). Parameters We run both Evosuite and Randoop on the ground truth program with 30 different seeds according to  (Arcuri and Briand, 2011) , we consider 30 runs in line with previous studies  Le et al.
Patches We have collected a total of 638 patches from 14 APR systems. All pass the sanity checks described in subsection 3. 6. Table  4  presents this dataset of patches for Defects4J. The first column specifies the dataset category and the second column gives the name of the automatic repair system. The number of patches collected per project of Defects4J is given in the third to the seventh columns and they are summed at the last column. They are 257 patches previously claimed as correct, which form D correct. There are 381 patches that were considered as overfitting by manual analysis in previous research, they form D overf itting. We found 160/257 patches from D correct are syntactically equivalent to the human-written patches: the exact same code modulo formatting, and comments. The remaining 97/257 patches are semantically equivalent to human-written patches.
Overall, the 638 patches cover 117/357 different bugs of Defects4J. 4 To our knowledge, this is the largest ever APR patch dataset with manual analysis labels by the researchers. The most related dataset is from  (Xin and Reiss, 2017a)  containing 89 patches from 4 repair tools and the one from  (Xiong et al. , 2018)  containing 139 patches from 5 repair tools. Our dataset is 4 times larger than the latter.
For Chart, Lang, Math and Time projects, RGT Evosuite2019 test suites achieve high code coverage on the buggy classes: the 90% percentile is higher than 80%. On the contrary, the code coverage by RGT Randoop2019 is clearly lower. The reasons are as follows: First, RGT Evosuite2019 suffers from fewer test generation failures: among 3510 random test suite generations, Evosuite fails to produce RGT tests in 31 runs while Randoop fails in 1080 runs, which lead to a respectively a 0. 9% and a 30. 8% failure rate. Second, Evosuite applies a genetic algorithm in order to evolve test cases that maximize code coverage, which has been consistently shown to be better than Randoop  (Shamshiri et al. , 2015; Kifetew et al. , 2019). Notably, the code coverage on the Closure project is significantly lower than for the other four projects, both for RGT Evosuite2019 and RGT Randoop2019 test suites.
We found two reasons that explain that: 1) the Closure project requires   (Shamshiri et al. , 2015) , 14,932,884 in RGT RandoopASE15  (Shamshiri et al. , 2015) , and 62,513 in RGT EvosuiteEM SE18  (Yu et al. , 2018). By conducting a sanity check of those tests, we discard 2. 7%, 4. 7% and 1. 1% flaky tests. Compared with the newly generated RGT tests, more flaky tests exist in previous generated tests due to external factors such as version, date and time  (Shamshiri et al. , 2015)  (#F9). To our knowledge, this is the largest ever curated dataset of generated tests for Defects4J.
Further, we have performed 30 executions of RGT tests and DiffTGen over the whole 381 patches from D overf itting. RGT Evosuite2019 yields 7,923 test fail- Figure  2  shows the number of overfitting patches in D overf itting dataset identified by RGT assessment and DiffTGen. RGT gives better results than DiffTGen for all Defects4J projects. An outlier case is the Closure project, for which we see that the assessment effectiveness is low, both for RGT (9/37) and for DiffTGen (0/37). This is consistent with the results as shown in.
DiffTGen does not configure a timeout for executing the body of a single test. On the contrary, RGT configures a timeout to bound the experimental time. As shown in Figure  4 , no overfitting patch is identified by RGT Evosuite2019 with the timeout difference (e. g. , D timeout ). In other words, the timeout difference setting has no influence on the experiment results. Thus, the experimental evaluation can be considered fair. DiffTGen and RGT Evosuite2019 differ in one key parameter: assertion generation. DiffTGen configures the assertion as false in Evosuite because it does not compare the behavior based on the oracles generated by Evosuite but based on the variables observed via monitoring with code instrumentation. Recall that to determine a patch's correctness, DiffTGen compares the values of instrumented variables between the patched version and the human-written version.
6 Second, the key novelty of our experiment is that we consider Evosuite which is not used in  Le et al. (2019). In our experiment, DiffTGen and Randoop respectively achieve the effectiveness of 37. 5%, 31%, while Evosuite reaches 65. 1%. This is a major result compared to  Le et al. (2019) : it shows that automated patch assessment is actually effective, which is essential for future progress in program repair. Finally, we suggest that different test generation tools can be used in combination, which is a pragmatic approach for practitioners: our study shows that Evosuite and Randoop put together in RGT can achieve a 72% effectiveness in identifying overfitting patches. Answer to RQ3: Out of the 381 patches claimed as overfitting by manual analysis, RGT assessment automatically identifies 274 / 381 of them (72%).
This is a significant improvement over  Le et al. (2019)  which reported that fewer than 20% of overfitting patches could be identified. RGT improves the state-of-the-art technique DiffTGen by 190% (274 versus 143 patches detected as overfitting), which strongly signals that RGT can reliably alleviate researchers to manually label overfitting patches. Our experiment is notable by its scale: the most related experiments are  Le et al. (2019)  and  Xin and Reiss (2017a)  based on 135 and 79 overfitting patches respectively, as opposed to 381 in this study (thus our experiment is performed on a 2. 8X and 4. 8X larger dataset).
Result of RQ4: Time Cost of RGT Patch Assessment Table  8  summarizes the time cost of RGT patch assessment. The first column gives the breakdown of time cost as explained in subsection 3. 5. The second and third columns give the cost for the RGT tests we have generated for this study, while the fourth to sixth columns are the three categories of RGT tests generated in previous research projects shared by their respective authors. T CGen time is not available for the previously generated RGT tests. They were reported by their authors but it is not our goal, thus we put a '-' in the corresponding cells. For example, the second column indicates RGT Evosuite2019 required 136. 3 hours for test case generation, 2. 9 hours for performing the sanity check, and 6. 2 hours for assessing the correctness of patches in D correct dataset and 9.
1 hours in D overf itting dataset. We observe that T CGen is the dominant time cost of RGT patch assessment. RGT Evosuite2019 and RGT Randoop2019 respectively spend 136. 3/154. 5 hours (88. 2%) and 109. 7/125. 1 hours (87. 7%) on test generation (#F10). For assessing 638 patches using newly generated RGT tests, we need 14. 5 minutes and 11. 76 minutes per patch for Evosuite and Randoop respectively. The three sets of previously generated RGT tests require 5. 2, 15. 3 and 5. 1 hours in accessing patch correctness for D correct and D overf itting dataset. Our experiment presents reusing tests from previous research is a significant time saver. For assessing 638 patches using previously generated RGT tests, the assessment time is 2. 4 minutes per patch on average. Note that the execution time of RGT EvosuiteASE15 is less than RGT Evosuite2019.
Therefore, RGT tests generated from previous research are considered effective and efficient for patch correctness assessment usage. Regression test generation is known to be costly and indeed, over 87% of the time cost in our experiment is spent in test generation. Consequently, reusing previously generated RGT test cases is a significant time-saver for patch assessment. By sharing a curated dataset of 4 million generated RGT tests, we save 246 computation hours for future researchers (not counting the associated time such as configuration, cluster management, etc). More importantly, reusing tests is essential for the scientific community: when experiments and papers are based on the same set of generated tests, the results can be reliably compared one against the others. Consequently, our replication dataset helps the community to achieve well-founded results.
Actionable Data Table  1  at the beginning of this paper lists the actionable implications obtained with our original experiments. Furthermore, our work provides actionable data for future research in automatic program repair. A dataset of 638 APR patches for Defects4J. We have collected and canonicalized 638 original patches from 14 different repair systems that form our experiment dataset. All patches have gone through strict sanity checks. This is a reusable asset for future research in program repair in particular to study anti-overfitting techniques and behavioral analysis. A dataset of 4,477,707 RGT tests for Defects4J. We have curated 4,477,707 generated test cases from two test generation systems. They complement the manual tests written by developers with the new assertions and input points sampled from the input space.
On the Comparison of PATCH-SIM Now, we present data analysis to compare the effectiveness of RGT with PATCH-SIM, the comparison is made on the same patches from the common repair systems. In their experiment and our experiment, there are a total of 20 overfitting patches in common: 5 from ACS, 6 from JGenProg2015, 6 from Nopol2015, and 3 from HDRepair. Our dataset does not overlap much with the PATCH-SIM's dataset. This is because we consider 14 repair systems Figure  6  presents the intersection of detected overfitting patches between RGT and PATCH-SIM from these four repair tools as Venn diagrams. The light gray indicates the overfitting patches identified only by RGT while the dark gray color indicates overfitting patches identified only by PATCH-SIM, the middle circles present the overfitting patches identified by both techniques.
2) we provide novel experiments to comprehensively study automatic patch correctness assessment, including false positive measurement, time cost estimation, and trade-off analysis; 3) they consider patches generated by 8 repair systems while we consider 14 repair systems; 4) their dataset is composed of 189 patches while our dataset contains 638 patches. Ye et al. (2020)  use RGT tests to access patch correctness on QuixBugs benchmark. There are two major differences in our experiment: (1) their experiment is performed on small buggy programs in which the total amount of lines of code ranges from 9 to 67 lines. Our experiment is performed on real-world bug repositories. (2) their dataset is composed of 64 patches while our dataset contains 638 patches. There are several works focusing on alleviating overfitting patch generation from the perspective of practical usage, which is not an automatic patches correctness assessment for scientific study.
Xiong et al. (2018)  propose PATCH-SIM and TEST-SIM to heuristically determine the correctness of the generated patches without oracles. They run the tests before and after patching the buggy program and measure the degree of behavior change. TEST-SIM complements PATCH-SIM by determining the test results of newly generated test inputs from Randoop. Our experiment shows Evosuite outperforms Randoop to sample test inputs to differentiate program behaviors. This suggests the effectiveness of this approach could be improved by also considering Evosuite for test generation. Although PATCH-SIM is able to filter out overfitting patches, we consider RGT assessment is better than PATCH-SIM for scientific study for two reasons: (1) RGT assessment achieves better effectiveness of identifying overfitting patches (72% of RGT and 56% of PATCH-SIM reported in  Xiong et al.
Our results are encouraging news for researchers in the program repair community: automatically generated test cases do help to assess patch correctness in scientific studies. To support the community and encourage automated patch assessment in future program repair experiments on Defects4J bugs, we make the dataset of 638 patches and 4,477,707 generated tests publicly available. As future work, we will consider goal-directed test generation to cover the changes in patches. Also, it is promising to consider techniques on regression testing minimization, selection and prioritization  (Yoo and Harman, 2012)  in order to speed up patch correctness assessment. With such techniques, it would be possible to identify redundant test cases and to remove them from the RGT test suite, to prioritize test cases, in order to maximize the likelihood of detecting a behavioral difference in a given amount of time.
Figure2shows the number of overfitting patches in D overf itting dataset identified by RGT assessment and DiffTGen. RGT gives better results than DiffTGen for all Defects4J projects. An outlier case is the Closure project, for which we see that the assessment effectiveness is low, both for RGT (9/37) and for DiffTGen (0/37). This is consistent with the results as shown in Figure1: RGT tests generated for the Closure project have the lowest coverage. As a result, the sampled RGT tests are less effective in exposing behavioral differences in the Closure project.
Case studies In the following sections we present three synthetic case studies that demonstrate the application of our approach to both extensional and contractional systems. All of the case studies have been implemented in the open-source 3D structural modelling library, LoopStructural 1. 2. 0 (https://doi. org/ Geosci. Model Dev. , 14, 6197-6213, 2021 https://doi. org/10. 5194/gmd-14-6197-2021 10. 5281/zenodo. 5234619, Grose et al. , 2021b, a). The case studies are provided as interactive Jupyter notebooks in https: //doi. org/10. 5281/zenodo. 5234634 (Grose, 2021).
The THOLE multiplet code The THOLE multiplet program is a suite of self-contained codes that have been adapted and modified by Theo Thole and his many coworkers. The first usage of the THOLE multiplet code was in the calculation of the 3d XAS spectra of the 4f rare earths  [15]. A multiplet calculation from 4f 6 to 3d 9 4f 7 has 48048 fin. l states, implying that the matrix diagonalizations become very large. These atomic calculations were performed using the Cowan code  [16] , which computes the reduced matrix elements of the atom in spherical symmetry. This needs only a few effective parameters: (a) the Slater integrals to describe the core-valence and valence-valence electron-electron interactions in the atomic potential and (b) the core and valence spin-orbit interaction  [17, 18]. The THOLE multiplet code is comprised of several self-contained codes that are run in sequence.
The RCN program calculates the initial and final-state wave functions in intermediate coupling using the atomic Hartree-Fock method with relativistic corrections, together with the Coulomb and exchange integrals (F k and G k ) and spin-orbit parameters. After empirical scaling of the Slater integrals, Cowan's RCG program calculates the electric dipole and quadrupole transition matrix elements from the initial state to the final-state levels of the specified configurations  [17]. Since the Cowan code uses spherical wave functions and Wigner-Racah tensor operators, which specify the magnetic quantum number, the Wigner-Eckart theorem can be used to obtain the spectra for linear and circular polarization in the presence of a Zeeman field  [19]. The next program of the THOLE code is based on Butler's point group program  [20].
Starting from the atomic reduced matrix elements, all the required reduced matrix elements can be obtained in any of the 32 different point groups, using coefficients calculated from group theory. The advantage of the Butler method, with respect to older methods  [21]  is that the coupling coefficients (isoscalar factors) are fully consistent over all point groups. The program can calculate the transition probabilities between any two configurations, such as the 2p XAS spectra of 3d transition metal ions. This version of the THOLE codes effectively treats crystal field theory, and over the last 35 years many calculations have been published using this approach  [15, 22, 23, 8, 9]. The program BANDER, developed by Thole in collaboration with Kotani and coworkers, extends the crystal field multiplet calculations with the hybridization between different configurations.
The number of configurations is only limited by the memory size and computing speed. This enables the study of the interplay between atomic multiplet structure and solid-state hybridization using the Anderson impurity model. The BANDER program exists in a number of varieties based on either exact diagonalization or the Lanczos method. The Lanczos method speeds up a large matrix calculation, by effectively describing the full spectrum by a specific limited set of levels. This speeds up the calculation, but if the set is chosen too small it has the disadvantage that specific final states are lost, which can lead to inaccuracies for second-order processes such as RIXS. Temperature dependent spectra are obtained taking a Boltzmann distribution over the calculated energy levels of the initial state.
The XCLAIM multiplet code The XCLAIM code makes use of an uncoupled product basis of atomic or real orbitals. In fact, the use of a product basis allows one to go one step further and make a code where the starting point is the Hamiltonian and not the basis. The code only needs to know how different "shells" are coupled via the one-and two-particle interactions in the Hamiltonian without any knowledge of atomic orbitals. A shell is defined as a set of states that are grouped together. The shell can contain atomic orbitals of a particular angular momentum, a subset of orbitals, spins, etc. The total system is then a combination of those shells. Physically, these shells can be on the same site or different sites, although the code only needs to know the coupling within and between the shells via the matrices for the one-and two-particle interactions.
Such a second-quantization based approach was initially incorporated in conventional programming languages, such as Fortran (as used in the underlying code for XCLAIM) and C, and later extended to script-based languages, such as Mathematica. In addition, to treating atomic spectroscopy, this approach allows one to build arbitrary clusters with the major restriction being the size of the many-body Hamiltonian. The Hamiltonian in matrix form is solved using Lanczos and tridiagonalization methods. The code itself is run by an external script that allows one to perform different types of calculations without changing the code. Finally, the code also calculates expectation values of importance for comparing X-ray spectroscopy with the sum rules for integrated intensities [Thole & van der Laan, 1988;  [24, 25].
One of the first advantages of this approach was the ability to study X-ray spectroscopy on systems containing more than one transitionmetal ion by making use of the flexibility in constructing systems and the possibility of using a reduced basis. This allowed the study of nonlocal screening effects in 2p core-level photoemission absorption site  [26] , the effects of doping  [27] , and the influence of exchange interactions between neighboring sites  [28]. The graphical user interface for XCLAIM, discussed in section 4. 2, provides a user-friendly way to create the input and script file for the underlying Fortran code and subsequently broadens and displays the output.
The QUANTY multiplet code This section reviews the empirical multiplet calculations using QUANTY  [29] [30] [31]. Parameter free calculations using QUANTY are described in section 4. 1. Philosophy: QUANTY implements a script language that can solve general problems in quantum chemistry and physics, focusing on spectroscopy and dynamics of correlated electron systems. The language allows one to define operators in second quantization and calculate Eigenstates or (generalized n-point) Green's functions for these operators. In principle one is completely free to define any operator and calculate any response function, given the memory and time constraints set. The idea behind the script language is that the user can focus on the physics whereas the numerical considerations are dealt with internally with only one parameter to control convergence, namely the required accuracy of the calculation.
It is common to take the multipole part of the Coulomb interaction from scaled atomic calculations, which works quite well as the multipole part of the Coulomb interaction is hardly screened in a solid. The monopole part is strongly screened and thus is treated as an empirical parameter. Implementation: The start of any QUANTY calculation is the definition of the basis orbitals used. Although the program in principle does not need this, it gives a physical meaning to the different Fermion indices or quantum numbers. For a ligand field calculation one could define that index 0-5 refers to the 2p shell, index 6-15 to the 3d shell and index 16-25 to the ligand shell, that the even (odd) indices refer to states with spin down (up) and that the atomic shells are given on a basis of spherical harmonics with the angular momentum quantized in the z direction ordered from -l to l.
Additional multiplet codes In addition to the THOLE, TANAKA, XCLAIM and QUANTY multiplet codes a number of additional codes have been written. Crocombette wrote a cluster-based charge transfer multiplet code, using the Slater integrals and spin-orbit coupling plus the Slater-Koster hopping, Hubbard U, charge transfer as parameters. No symmetry restrictions have been used. The CROCOMBETTE multiplet code has been used to calculate the 2p XAS of 3d transition metal oxides such as TiO 2 and Fe 2 O 3 , with equivalent results to the THOLE calculations  [12]. Stepanow et al. have written a charge transfer multiplet code that is based on Cowan, but that is not limited by symmetry. The STEPANOW multiplet code is a single-atom symmetry-free multiplet code that is mainly used to calculate XMCD spectra and spin-and orbital expectation values related to 2p XAS spectra of 3d systems and 3d XAS spectra of rare earths  [33].