dataesr/scientific-paragraphs-categorization huggingface dataset:
Sentence-piece tokenization divides every token into a sequence of subtokens. Inference Phase. In the inference phase, we use the trained model to repair future bugs from the same project. Specifically, SelfAPR takes an input of a buggy program (P) and the failing test diagnostic D. We employ fault localization (FL) (e. g. , Ochiai  [1]  or Gzoltar  [57] ) to generate a ranked list of suspicious buggy lines. For a given suspicious statement found by FL, SelfAPR constructs an inference input per our representation with: 1) the suspicious statement and its context, and 2) the test execution diagnostic. This input is given to the patch generator (transformer-based neural model), which outputs the most likely patch, or may enumerate the 𝐾 best patches for that suspicious statement with beam search, where 𝐾 is fully configurable, a.
k. a, the beam search size. To our knowledge, our work is novel in two aspects. First, it is the first to propose project-specific training using a different version of the same project under repair. This enables the neural model to learn project-specific knowledge and mitigate the training discrepancy when the training and testing datasets come from different projects. Second, SelfAPR is the first to add compiler and test suite execution information as part of the input representation for neural program repair. As an opposite, the previous supervisedlearning studies all consider as input a pair of the buggy and fixed source code. The related two self-supervised learning approaches  [3, 79]  neither include test execution diagnostics, nor are founded in project-specific training with a past version of the project under repair.
Perturbation Model The goal of the perturbation model is to generate training samples with bugs. Recall that the perturbation model generates bugs from a correct program. We train the neural model with the buggy programs as input and learn to generate the correct version as output. In SelfAPR, we design the perturbation rules (𝑅𝑢𝑙𝑒𝑠) driven by learning objectives that we expect the neural model to learn. For example, if the learning objective is to learn how to insert statements, a perturbation model may generate training samples by deleting statements: reversed, the training sample teaches the neural model to learn inserting code that has been deleted.
𝑅𝑢𝑙𝑒 7-2 replaces the correct invocation with a new one that appears in the class file. For both rules, SelfAPR synthesizes the arguments following the typing and scoping constraints of available variables. 𝑅𝑢𝑙𝑒 8 generates a compound statement by stacking 𝑅𝑢𝑙𝑒 1 -𝑅𝑢𝑙𝑒 7 from a correct statement in order to increase the complexity of the learning task for some training samples. Learning objective: learning to reuse existing code from the same program. Ever since GenProg  [36] , it is known that reusing code from the program repair is valuable  [4, 44]. Hence, we want perturbations that encourage the neural network to reuse code from in the close vicinity of the buggy location. 𝑅𝑢𝑙𝑒 6 are dedicated to boolean expressions. 𝑅𝑢𝑙𝑒 6-1 perturbs boolean expressions by removing a clause. 𝑅𝑢𝑙𝑒 6-2 expands a correct boolean expression by transplanting an existing binary expression in the method scope, encouraging clause reuse.
𝑅𝑢𝑙𝑒 9 modifies the correct statement by transplanting a similar statement taken from the class scope, called the donor statement. In SelfAPR, the donor statements are selected based on edit distance with the target statement, and statements with a higher similarity score than the default threshold are selected. Notably, code transplantation is a more generic strategy than the others, it augments the diversity of the generated training samples behind the transformations by the previous 𝑅𝑢𝑙𝑒𝑠. Learning objective: learning to synthesize code according to the context. It is known that code has low entropy, because of its high contextual redundancy  [25]. We design the following perturbations in order to for the neural network to learn to synthesize a patch according to the closest surrounding code.
Diagnostic Extraction SelfAPR generates training samples that are guaranteed to be buggy (recall Section 3. 2). For each perturbed program P ′ , SelfAPR executes it against the compiler and available test suite to determine whether it is a valid buggy training sample. Next, SelfAPR extracts a diagnostic D of the error. The diagnostic may be a compiler error diagnostic (𝐷 𝑐 ) or a test failure diagnostic (𝐷 𝑡 ). If a perturbed program does not compile, 𝐷 𝑐 is the compiler error message. If a perturbed program P ′ is compilable, then P ′ is executed against the available test suite. If one test case fails, this results in a test execution diagnostic (𝐷 𝑡 ). This diagnostic extraction is a key novelty of our work. The intuition is that they provide signals related to the bug type. The diagnostics enable the neural patch generator to generate patches according to tokens of the error diagnostic.
Implementation We implement our perturbation model based on Spoon  [53]  which consists of 85 functions and more than 5K lines of code. We implement SelfAPR's patch generator with state-of-the-art Transformer based architecture  [56]  from HuggingFace. The encoder and decoder consist of 6 layers. We configure SelfAPR to take a maximum of 384 input tokens from buggy and context code and generate a patch with a maximum of 76 tokens. SelfAPR is trained for 10 epochs, using a batch size of 32 and a vocabulary size of 32 128.
Training and Testing Datasets We construct a dataset of programs P to seed the self-supervision loop. Recall that SelfAPR evaluates the perturbation-based training samples with compiler and test suite. This forces the correct programs P to meet the following two requirements: 1) P needs to be buildable in order to capture the compiler error diagnostics. 2) P needs to have a test suite, that can be automatically executed using a test driver, in order to capture test execution failure diagnostic. Consequently, we look for benchmarks that comply with this hard compilation and testing constraints. We chose to use the widely accepted Defects4J  [32]    Training sample generation. Table  1  shows the details of our training and testing sets. The first column gives the name of the open-source project, and the second to the eighth columns give the details of the training data including the number of perturbationbased samples generated.
The last two columns give information about the testing set, including starting date of the bugs and the number of bugs. For example, the first row shows that for the Closure project, we use the source code in commit 6d374c3 from 2010-01-08 for generating perturbation-based training samples, which is composed of  60    Perturbation analysis. Figure  4  shows the distribution of the perturbation-based samples generated with SelfAPR's perturbation rules. From this figure, we make the following observations: 1) All perturbation rules contribute to generating training samples and result in a diversity of training samples. 2) Not all the rules equally generate samples. This is explained by the characteristics of the code under the perturbation. For example, the variable perturbation 𝑅𝑢𝑙𝑒 11 leads to the most training samples while the perturbation 𝑅𝑢𝑙𝑒 12 yields the least (wrap the target statements with a block).
Methodology for RQ1 In RQ1, we compare SelfAPR against the state-of-the-art of 1) supervised learning repair approaches: SequenceR  [13] , CoCoNuT  [42] , CURE  [31] , Recoder  [85] , and RewardRepair  [83]. We do not include TFix  [6]  because it is trained on JavaScript, which cannot be used to evaluate Defects4J bugs in Java. 2) Semantics-based repair approaches: Nopol  [76]  and DynaMoth  [18] ; 3) Search-based repair approaches: GenProg-A  [84]  (the Java implementation of Genprog  [36] ), SimFix  [30]  and TBar  [37]. Moreover, we re-implement the Java version of the selfsupervised learning approach of BugLab  [3]  (originally designed for Python) with all four perturbation rules regarding operators, variables and literals. Recall that the goal of BugLab is not to obtain project-specific training samples, thus it is not explicitly executed on a past version of the project under repair.
Methodologies for RQ2 & RQ3 In RQ2, we evaluate the effectiveness of SelfAPR with and without training samples from the project under repair. Recall that in RQ1, our training set and testing set contain the same projects, while the testing test comes from the latter commits to guarantee the fairness and the practical usage of SelfAPR (i. e. , make sure the fixes are not used during the training). Nevertheless, the training samples from a past version contain project-specific context, e. g. , code tokens and similar expressions. Consequently, in RQ2, we exclude project-specific training samples. We create one new training set per project by discarding perturbation-based samples from the project under repair. For example, to test SelfAPR's effectiveness on project Chart, we create a training set without the 225 953 training samples from Chart.
As seen in Table  2 , SelfAPR outperforms BugLab by a large margin. The reason comes from the perturbation model. The one of BugLab is narrow and restricted to specific bug types. On the contrary, SelfAPR's perturbation model considers more rules and they are generic in nature. This results in diversity and genericity of training samples. In particular, no code transplantations and no deletions are considered in BugLab. This not only decreases the chances of learning to insert and learning to delete, but also fails to generate more training samples with project-specific knowledge (usage of domain types and methods). Deeply integrated prioritization. SelfAPR outperforms the related work on search-based and template-based repair approaches. One key reason may relate to enumeration and prioritization.
Quantity versus quality of training samples. SelfAPR is trained on fewer samples than CoCoNuT, CURE and RewardRepair, yet yields better performance. This suggests the perturbation-based training samples are of higher quality and contain more information. There are two reasons for it: project-specific knowledge and no noisy commits. For supervised program repair, the past commits used for training from GitHub suffer from many limitations: they are not guaranteed to be atomic bug fix commits  [62]  and they may include unrelated code changes (e. g. , new functions, comments and optimization, etc). On the contrary, all training samples generated by our perturbation model are controlled and guaranteed to bugtriggering samples with no unrelated changes. Uniquely repaired bugs. Compared with all learning-based repair approaches, SelfAPR uniquely repairs 10 bugs that have never been reported as repaired by other learning-based approaches, while the other three approaches CURE, Recoder and RewardRepair also uniquely repair respectively 6, 9, and 2 different bugs.
Threats to Validity An internal threat relates to 1) our implementation of the perturbation model may contain bugs that could prevent generating more appropriate perturbation-based training samples and 2) manual patch correctness assessment. To mitigate these threats, we make the perturbation tool and generated patches publicly available for further assess with related techniques  [63-65, 74, 80]. A threat to external validity relates to whether the performance of SelfAPR generalizes to arbitrary programming languages. Per the standards of the field, our approach has been tested in one language (Java) and the evaluation is carried out on well-established benchmarks. In principle, our approach can be applied to other programming languages and datasets.
Multi-location Bugs Repairing bugs spread in different locations (e. g. , multi-hunk bugs) remains challenging for the program repair community. Our work SelfAPR, as much as the related work does not succeed in repairing the 420/818 multi-location bugs in Defects4J. The complexity of repairing multi-location bugs not only comes from the fixing tokens and expressions, but also from the interaction between fixes in different locations. To our knowledge, only three prior work from semantics-based and search-based approaches target on multilocation bugs: Angelix  [47] , VarFix  [73]  and Hercules  [58].
Self-supervised Learning on Code Self-supervised learning based APR has been little explored. Loriot et al. [41]  devise a self-supervised learning loop to repair formatting issues. Yasunaga and Liang  [79]  propose self-supervised learning for repairing compilation errors. In both cases, the idea is to do character level perturbations (e. g. , replacing or deleting punctuation). On the contrary, we use AST level perturbations, which are a much larger scope and are more impactful. Only our AST level perturbation can trigger functional errors and learn to repair them. Allamanis et al. [3]  train a bug detection and repair model called BugLab. The key differences with our work are that they do not execute the perturbation-based programs, therefore, no test diagnostics are included into the input representation of a bug.
SelfAPR is the first to generate training samples in a project-specific manner with a past version of the project under repair. A line of work considers self-supervised learning for other downstream tasks than program repair, e. g. , code retrieval and code summarization  [2, 7, 8, 20]. The perturbation strategies employed are typically based on token masking or single token perturbation. None of those previous works involves a perturbation model at the AST level that needs to respect strict constraints of programming languages (e. g. , variable scopes) as SelfAPR does.
CONCLUSION We present a novel neural program repair model called SelfAPR, which introduces two major novelties wrt the related work: First, it uses self-supervision with automatically generated training samples with program perturbation. Secondly, it adds execution information into the input representation that captures the failing assertion of the program under repair. Thanks to those two key features, SelfAPR repairs 110 out of 818 bugs from Defects4J. Notably, 10 of them were never repaired before by the supervised learning repair approaches, demonstrating the value and power of project-specific training and test diagnostics embedded.
• We perform an original series of experiments and show that SelfAPR repairs 110 bugs from 17 open-source projects. Our experimental results demonstrate that self-supervised learning on a past version of the program under repair signif-icantly increases the repair effectiveness. Our experiments show that including project-specific training samples directly contributes to repairing 20 more bugs (+30%) on Defects4J version 1. 2. • We consolidate and share our training dataset of 1 039 873 buggy training samples, including 408 858 functional errors with at least one failing test case and 631 015 compilation errors. This dataset is valuable for future researchers to un-derstand the syntactic and semantic relationships between errors and code changes responsible for them. We make all our code and data publicly available at https://github.
filtering. Note that our testing Defects4J bugs are all caused by function errors. To ensure the close distribution between training data and testing data, we conduct training dataset filtering as follows: 1) keep the [FE] samples and those [CE] samples with semantic errors, and 2) remove [CE] samples with syntactic compilation errors, e. g. , "; is expected ". Approaches Nopol [76] (no learning) DynaMoth [18] (no learning) GenProg-A [84] (no learning) SimFix [30] (no learning) TBar [37] (no learning) SequenceR [13] (supervised learning) CoCoNuT [42] (supervised learning) CURE [31] (supervised learning) RewardRepair [83] (supervised learning) Recoder [85] (supervised learning) BugLab [3] (self-supervised learning) SelfAPR (self-supervised learning) # Training -----35 578 24 471 491 24 471 491 2 307 241 103 585 415 687 1 039 873 # Beam -----50 1000 1000 200 100 50 50 Spectrum-based FL D4J (v1.
We present a new open-source data-reduction pipeline to reconstruct spectral data cubes from raw SPHERE integral-field spectrograph (IFS) data. The pipeline is written in Python and based on the pipeline that was developed for the CHARIS IFS. It introduces several improvements to SPHERE data analysis that ultimately produce significant improvements in postprocessing sensitivity. We first used new data to measure SPHERE lenslet point spread functions (PSFs) at the four laser calibration wavelengths. These lenslet PSFs enabled us to forward-model SPHERE data, to extract spectra using a least-squares fit, and to remove spectral crosstalk using the measured lenslet PSFs. Our approach also reduces the number of required interpolations, both spectral and spatial, and can preserve the original hexagonal lenslet geometry in the SPHERE IFS.
g. ,  Racine et al. 1999; Sparks & Ford 2002; Hoeijmakers et al. 2018). The recent generation of IFS instruments combines the high angular resolution of the telescope with extreme adaptive optics (XAO) to provide near diffraction-limited images of stars and their surroundings (e. g. ,  Guyon 2005; Jovanovic et al. 2015). The difference in brightness between the star and potential planets necessitates the use of further technologies such as coronagraphs ⋆ The pipeline and documentation are publicly available at https: //github. com/PrincetonUniversity/charis-dep (e. g. , Soummer 2005; Guyon et al. 2005; Snik et al. 2012)  to suppress the light from the central star. Dedicated high-contrast imaging techniques are also needed to distinguish between the unavoidable residual starlight halo and genuine astrophysical signals in the direct vicinity of the star, such as angular differential imaging (ADI;  Marois et al.
The reduction and processing of IFS data is inherently complex. After investing a large part of their budgets in the initial development and building of an instrument, individual teams therefore often struggle to provide the manpower for continuous support and improvement of the software that is used for data reduction. This can be remedied by providing the scientific community access to source codes and encourage community involvement in their further development and in improving their reliability. In this paper we present our effort to take the first step in this direction by adapting the CHARIS data-reduction pipeline  (Brandt et al. 2017, henceforth B17) , an open-source package written in Python and Cython available on Github, to be able to process SPHERE IFS data into scientifically usable products.
The calibration unit also takes wavelength calibration data by illuminating the lenslets with lasers of known wavelength. All of these data are taken during the day, both for calibration purposes and to monitor the health of the instrument. Dark and background images are likewise obtained by closing shutters at different positions (in front of the IFS module for darks, or in front of the SPHERE entrance shutter for instrument backgrounds) during daytime. This saves observing time during the night for actual science data. The calibration unit consists of (a) a series of lamps, both narrow and broad band, for taking detector flat-fields and flat-fields taken through the integral field unit (IFU), and (b) a series of monochromatic lasers for wavelength calibration. For SPHERE, the 'wavecal' calibration template that is routinely executed takes images with three (YJ-mode) or four (YH-mode) lasers simultaneously (see left panel, Fig.
1 ). The first step in the analysis of SPHERE IFS data is to take the 2D array of microspectra and reconstruct a 3D data cube: an image at each sampled wavelength. Subsequent analysis steps, usually referred to as postprocessing, subtract starlight to search for faint exoplanets, calibrate and extract spectra, and calculate contrasts and sensitivities. The ESO data-reduction and handling (DRH) pipeline  (Pavlov et al. 2008 ) is designed to perform all steps that are required to reconstruct data cubes from SPHERE IFS raw data. This includes everything from thermal background subtraction and flat-field correction to the more intricate steps of wavelength calibration and extraction of the spectra into usable spectral image cubes. The SPHERE Data Center 2  (DC,  Delorme et al. 2017) , as well as a third-party Python wrapper  (Vigan 2020) , both combine the DRH with additional functions and improvements.
Because of the relatively close spacing of the microspectra in SPHERE, a larger aperture also includes more flux from neighboring spectra at other wavelengths (spectral crosstalk). These shortcomings may be overcome by optimal extraction (Horne 1986), as long as the line-spread function perpendicular to the dispersion direction (the instrumental profile) is known. This has not previously been measured for SPHERE, preventing an implementation of more sophisticated spectral extraction approaches. The SPHERE instrument uses hexagonal lenslets that are arranged in a honeycomb pattern (for an example of this). The DRH resamples the microspectra onto a rectilinear grid (see also Fig. 8 ). This requires an interpolation and sacrifices some of the information in the raw data. Previous generations of highcontrast instruments lacked spectral information and took data on rectilinear grids (the arrangement of pixels on nearly all detectors).
The interpolation step taken by the DRH enables established postprocessing routines to work on SPHERE data without accounting for a new geometry. The use of aperture photometry of the SPHERE DRH to extract microspectra and its resampling of the data cubes onto a rectilinear grid suggest that a new extraction approach could result in higher-quality data cubes. Berdeu et al. (2020)  suggested an alternative regularized least-squares approach to reduce SPHERE IFS data, called PIC, and saw a reduction in data artifacts and improvements in final postprocessed contrast. In the following sections, we describe an independent data-reduction pipeline based on that of the CHARIS IFS  (Groff et al. 2016; B17). This approach revisits every aspect of the data reduction and also takes advantage of new calibration data.
Calibrations Calibration data for the SPHERE IFS consist of uniformly illuminated images of the detector for flat-fielding, dark images for background subtraction, and wavelength calibration frames. The latter consist of the lenslet array uniformly illuminated by three or four lasers of known wavelength fed through an integrating sphere. Dark images have significant counts because most of SPHERE is not cryogenic. Darks in SPHERE IFS images consist of two components: an undispersed background from scattered light and emission after the lenslet array, and a dispersed background that is imaged by the lenslet array into microspectra. Sky images include both backgrounds, while dark images include only the undispersed background. Our approach uses only the dark (thermal instrument background) images and the detector flat-fields.
From these, we aim to construct a detector flat-field, a bad-pixel mask, and an undispersed background template. All calibration files necessary to reduce SPHERE IFS data are packaged with our pipeline, except for the laser flat-field needed to adjust the wavelength solution for each observing night. We include these master calibrations rather than adopting nightly calibrations for two reasons. First, each individual calibration contains its own realization of read noise and photon noise. If nightly calibration frames are used, the same realization of noise is added to each science frame, potentially doubling the read and background noise. Master calibrations avoid this drawback, thereby providing better results as long as the calibrations themselves are stable. In the following sections, we demonstrate the requisite stability of both the background and the flat-fields.