dataesr/scientific-paragraphs-categorization huggingface dataset:
Target Domain Training Next, we use the source domain validation dataset to select the best model produced by source domain training, and tune it on a vulnerability fixes dataset. The intuition is that the knowledge from bug fixing can be transferred to the vulnerability fix domain. We follow the same procedure mentioned in the subsection 3. 4 to extract the buggy and fixed functions in the VRepair representation from all vulnerability fixes in the vulnerability dataset. We precede all inputs with a special token 'CWE-xxx' identifying the CWE category to which the vulnerability belongs. To ensure sufficient training data for each 'CWE-xxx' special token, we compute the most common CWE IDs and only keep the CWE IDs with sufficient examples in the vocabulary. The top CWE IDs that we keep cover 80% of all vulnerabilities and the CWE IDs that are not kept in the vocabulary are replaced with 'CWE-000' instead.
As we discuss in Section 3. 7, the Transformer model learns to produce outputs that are likely to be correct based on the training data it has processed. Beam search is a wellknown method  [26]  in which outputs from the model can be ordered and the n most likely outputs can be considered by the system. Beam search is an important part of inference in VRepair. We introduce in VRepair a novel kind of beam, which is specific to the code representation introduced in subsection 3. 2, defined as follows. Definition The Neural beam is the set of predictions created by the neural network only, starting with the most likely one and continuing until the maximum number of proposals configured has been output  [26]. Neural beam search works by keeping the n best sequences up to the current decoder state.
Hence, the interpretation beam is 3 for context size 2, and 1 for context size 3. Definition The VRepair beam is the combination of the neural beam and the interpretation beam, it is the cartesian product of all programs that can be created from both the Neural beam and the Interpretation beam. Once VRepair outputs a patch, the patched function is meant to be verified by a human, a test suit, or a static analyzer, depending on the software process. For example, if the vulnerability was found by a static analyzer, the patched program can be verified again by the same static analyzer to confirm that the vulnerability has been fixed by VRepair. As this evaluation may consume time and resources, we evaluate the output of the VRepair system by setting the limit on VRepair beam which represents the number of programs proposed by VRepair for evaluation.
Neural Network Architecture VRepair uses the Transformer architecture  [23]  as sketched in Figure  3. The Transformer for VRepair learns to repair vulnerabilities by first receiving as input the code with a vulnerability encoded in our data representation (subsection 3. 2). Multiple copies of multi-head attention layers learn hidden representations of the input data. These representations are then used by a second set of multi-head attention layers to produce a table of probabilities for the most likely token to output. In addition, we use a copy mechanism that trains an alternate neural network to learn to copy input tokens to the output in order to reduce the vocabulary size required  [22] ,  [38]. The first token to output is based solely on the hidden representations the model has learned to produce from the input code.
As tokens are output they are available as input to the model so it can adjust the next token probabilities correctly. For example, in Figure  3 , after the sequence of tokens '&& ( d > a * ' has been output, the model predicts that the next token should be 'b' with a probability of 0. 8. Libraries VRepair is implemented in Python using stateof-the-art tools. We download all GitHub events using GH Archive  [39]. For processing the source code we use GCC and Clang. Once the source code is processed, OpenNMTpy is used to train the core Transformer model  [40]. Hyperparameters Hyperparameters define the particular model and dataset configuration for VRepair and our primary hyperparameters are presented in Table  1. The context size is a hyperparameter specific to VRepair and is discussed in more detail in subsection 3.
Usage of VRepair As indicated by Figure  1 , VRepair is intended to be used for proposing vulnerability fixes within an environment that provides vulnerability detection. Recall that vulnerability detection is not in the scope of VRepair per se, tools such as Infer can be used to statically analyze code and indicate a suspicious line containing a vulnerability (see subsection 3. 2). Once a patch is generated, the expected use case is to recompile the function with the proposed vulnerability fix, to pass a functional test suite if such a test suite exists, and then to pass the vulnerability checker under consideration (such as Infer). As the last step, having.
EXPERIMENTAL PROTOCOL In this section, we describe our methodology for evaluating our approach by defining 4 research questions and how we propose to answer them. RQ1 will explore the performance of the neural model when only trained with the small dataset available from the target domain (vulnerability fixing), or only trained with a larger dataset from the bug fixing source domain; both being called 'single domain training'. After exploring single domain training, RQ2 will study the effectiveness of using transfer learning to mitigate the issue of the small vulnerability dataset size demonstrated in RQ1. For RQ2, we study whether a model trained on the source domain (bug fixing) and then tuned with the target domain (vulnerability repair), produces a better result than either source or target domain training alone.
Bug Fix Corpus We create a bug fix corpus by mining the GitHub development platform. We follow the procedures in related works  [26] ,  [46]  and collect our bug fixing dataset by filtering GitHub commits to C code projects based on keywords such as 'bug' or 'vulnerability' in the commit message. Filtering commits based on commit messages can be imprecise and generate false positives. However, this imprecision has minimal effect for our source domain dataset -any code change will likely help train the model on how to propose code patches. We download 892 million GitHub events from the GH Archive data  [39]  which happened between 2017-01-01 and 2018-12-31. These events have been triggered by a Github issue creation, an opened pull request, and other development activities. In our case, we focus on push events, which are triggered when at least one commit is pushed to a repository branch.
In total there were 478 million push events between 2017-01-01 and 2018-12-31. Next, we filter bug fix commits as follows. Per the related work  [26] ,  [46] , we adopt a keyword-based heuristic: if the commit message contains keywords (fix OR solve OR repair) AND (bug OR issue OR problem OR error OR fault OR vulnerability), we consider it a bug fix commit and add it to our corpus. In total, we have analyzed 729 million (728,916,054) commits and selected 21 million (20,568,128) commits identified as bug fix commits. This is a dataset size that goes beyond prior work by Tufano et al. [26] , who uses 10 million (10,056,052) bug-fixing commits. In our experiment, we focus on C code as the target programming language for automatic repair. Therefore we further filter the bug fix commits based on the file extension and we remove commits that did not fix any file that ends with '.
c', resulting in 910,000 buggy C commits. For each changed file in each commit, we compare all functions before and after the change to extract functions that changed; we call these function pairs. To identify these function pair changes, we use the GNU compiler preprocessor to remove all comments, and we extract functions with the same function signature in order to compare them. Then, we used Clang  [47]  to parse and tokenize the function's source code. Within a '. c' file we ensure that only full functions (not prototypes) are considered. In the end, we obtain 1,838,740 function-level changes, reduced to 655,741 after removing duplicate functions. The large number of duplicates can be explained by code clones, where the same function is implemented in multiple GitHub projects. As detailed in Section 3.
Vulnerability Fix Corpus We use two existing datasets called Big-Vul  [15]  and CVEfixes  [16]  for tuning the model trained on the bug fixing examples. The Big-Vul dataset has been created by crawling CVE databases and extracting vulnerability related information such as CWE ID and CVE ID. Then, depending on the project, the authors developed distinct crawlers for each project's pages to obtain the git commit link that fixes the vulnerability. In total, Big-Vul contains 3754 different vulnerabilities across 348 projects categorized into 91 different CWE IDs, with a time frame spanning from 2002 to 2019. The CVEfixes dataset is collected in a way similar to the Big-Vul dataset. This dataset contains 5365 vulnerabilities across 1754 projects categorized into 180 different CWE IDs, with a time frame spanning from 1999 to 2021.
Methodology for RQ2 The state-of-the-art vulnerability fixing models are usually trained on a relatively small vulnerability fixing dataset  [6] , or generated from synthesized code examples  [5]. Based on prior studies showing the effectiveness of large datasets for machine learning  [8] , we hypothesize that it is hard for a deep learning model to generalize well on such a small dataset. In this research question, we compare the performance between the transfer learning model and the model only trained on the small vulnerability fix dataset. We first train the models with source domain training, i. e. , we train them on B train and apply early stopping on B val. The models from source domain training are used in the target domain training phase. We continue training the models using Big-Vul rand train (or CVEfixes rand train ) and the new model is selected based on Big-Vul rand val (or CVEfixes rand val ) with early stopping.
Methodology for RQ3 We compare our source domain training with the state of the art pre-training technique of PLBART  [50]  on denoising. This means we first train on an unsupervised task (the denoising task). The trained model is then finetuned with target domain training as we do for VRepair. We hypothesize that initial training on a related task is more helpful than initial training on a generic unsupervised task. Specifically, we select full functions from our bug fix corpus and apply PLBART's noise function techniques for token masking, token deletion, and token infilling. This results in artificial 'buggy' functions whose target repair is the original function. As per PLBART (and the original BART paper addressing natural language noise functions  [51] ): token masking replaces tokens with a special token <MASK> in the 'buggy' function; token deletion removes tokens; and token infilling replaces multiple consecutive tokens with a single <MASK> token.
As in PLBART, we sample from a Poisson distribution to determine the length for the token infilling noise function. Following this methodology, we generate the pretraining dataset from the bug fix corpus divided into Pre train (728,730 samples) and Pre val (10,000 samples). For the pre-trained model, similar to subsection 4. 4, we first train the models on Pre train and apply early stopping on Pre val. We continue training the models using Big-Vul rand train (or CVEfixes rand train ) and the new model is selected based on Big-Vul rand val (or CVEfixes rand val ) with early stopping. Finally, the final model is evaluated with Big-Vul rand test (or CVEfixes rand test ), and compared against the transfer learning models trained in subsection 4. 4.
RQ3 studies the effect of replacing the source domain training phase of transfer learning with denoising as pretraining in VRepair. As explained in subsection 4. 5, we consider the state-of-the-art pre-training technique from PLBart  [50]. In Table  10   From a qualitative perspective, denoising pre-training has the advantage of being unsupervised and therefore does not require collecting and curating a source domain dataset. Thanks to this property, CodeBERT  [53] , CuBERT  [54]  and PLBART  [50]  all have millions of examples in the pretraining dataset. On the other hand, He, Girshick, and Dollár found that the performance of a pre-trained model scales poorly with the pre-training dataset size  [55]. This result shows that even when the size of the source domain dataset (i. e. , B train and B val ) is slightly smaller than the size of the pre-training dataset (i.
Results for RQ3 Answer to RQ3: In this experiment, transfer learning outperforms denoising pre-training and finetuning with datasets of similar size (21. 86% versus 13. 36% on Big-Vul rand test , and 22. 73% versus 13. 34% on CVEfixes rand test ). This result shows that the effort of collecting and curating a source domain dataset, arguably a tedious and consuming task, pays off with respect to performance. Overall, the specific source domain task of bug fixing is better than the generic task of denoising.
Results for RQ4 In RQ4, we explore different ways of creating test datasets, each of them capturing an important facet of transfer learning. Table  12 , Table  13 , Table  14  and Table  15  show the test sequence accuracies for all considered data splitting strategies. For each table, the first column lists the top-10 most common CWE IDs in each data split of the respective dataset. The second column shows the performance of the transfer learning model, which is a model trained on the large bug fix corpus, and then tuned with the vulnerability fix examples. The third column presents the performance of the model trained with the small dataset only, i. e. , only target domain training on each training split of different data splits. The last row of each table presents the test sequence accuracy on the whole test set on each data split.
From all tables, we can clearly see a large difference between the performance of transfer learning and target Interestingly, the performance of models trained with target domain training only varies a lot between different data split strategies. It varies from 0. 13% in CVEfixes year test on the CVEfixes dataset to 11. 29% in CVEfixes rand test. In other words, the performance of a vulnerability fixing system trained on a small dataset is unstable; it is highly dependent on how the data is divided into training, validation, and testing data. This has been observed in prior research as well  [12] : the knowledge learned from a small dataset is often unreliable. On the other hand, transfer learning models have stable performance, staying in a high range from 16. 23% on CVEfixes year test to 22.
73% on CVEfixes rand test. When comparing the test sequence accuracy for each vulnerability type, i. e. , different CWE IDs, the transfer learning models also surpassed models trained only on the target domain. The only exception is CWE-190 in CVEfixes rand test. It may be that the nature of the fixes for the CWE varies over time such that transfer learning was more beneficial for the Big-Vul year test and CVEfixes year test splits. When comparing the overall performance on Big-Vul rand test and CVEfixes rand test , target domain training versus transfer learning, the transfer learning model is still to be preferred. We argue that splitting the vulnerability fix dataset based on time is the most appropriate split for evaluating a vulnerability repair system. By splitting the dataset based on time and having the newest vulnerabilities in the test set, we simulate a scenario where VRepair is trained on past vulnerability fixes and evaluated on future vulnerabilities.
To this extent, we suggest that Big-Vul year test and CVEfixes year test are the most representative approximations of the performance of VRepair in practice. Answer to RQ4: For the two considered data splitting strategies (random and time-based) transfer learning achieves more stable accuracies (16. 23% to 22. 73% for transfer learning, and 0. 13% to 11. 29% for models only trained on the small vulnerability fix dataset). This validates the core intuition of VRepair: transfer learning overcomes the scarcity of vulnerability data for deep learning and yields reliable effectiveness.
ABLATION STUDY During the development of VRepair, we explored alternative architectures and data formats. To validate our explorations, we perform a systematic ablation study. Table  16  highlights 8 comparisons that are of particular interest. The description column explains the way the model was varied which produced the results. All the models in the ablation study are evaluated on the Big-Vul dataset. ID 0 is our golden model of VRepair. We include it in the table for easier comparison against other architectures and dataset formats that we have tried. ID 1 summarizes the benefit of using beam search from the neural network model for our problem. In this comparison, the pass rate on our target dataset Big-Vul rand test increased from 8. 96% with a single model output to 21. 86% with a beam size of 50.
9% test sequence accuracy. However, most static analysis tools will not provide this information. For example, Infer  [56]  only provides a single vulnerable line as output for each found vulnerability. If we limit our model to only predict changes for a contiguous block of lines (i. e, one or more lines after the erroneous line is identified), then the test sequence accuracy is 28. 39%. We note that single block repairs (which include single-line repairs) form 57. 84% of our dataset, so IDs 4 to 7 show our ablation of the key hyperparameters in our golden model. ID 8 shows why our vocabulary size is set to a rather low value of 5000, which is done thanks to using the copy mechanism. We clearly see that by using the copy mechanism, the model can handle the out-ofvocabulary problem well with a low vocabulary size of 5000.
Vulnerability Fixing with Learning We include related work that fixes vulnerabilities with some kind of learning, meaning that the system should learn fix patterns from a dataset, instead of generating repairs from a set of pre-defined repair templates. Vurle is a template based approach to repair vulnerability by learning from previous examples  [9]. They first extract the edit from the AST diff between the buggy and fixed source code. All edits are then categorized into different edit groups to generate repair templates. To generate a patch, Vurle identifies the best matched edit group and applies the repair template. Vurle is an early work that does not use any deep learning techniques and is only trained and evaluated on 279 vulnerabilities. In contrast, VRepair is trained and evaluated on 3754 vulnerabilities and is based on deep learning techniques.
Harer et al. [5]  proposed using generative adversarial networks (GAN), to repair software vulnerabilities. They employed a traditional neural machine translation (NMT) model as the generator to generate the examples to confuse the discriminator. The discriminator task is to distinguish the NMT generated fixes from the real fixes. The trained GAN model is evaluated on the SATE IV dataset  [57]  consisting of synthetic vulnerabilities. Although Harer et al. trained and evaluated their work on a dataset of 117,738 functions, the main limitation is that the dataset is fully synthetic. In contrast, our results have better external validity, because they are only based on real world code data. SeqTrans is the closest related work  [6]. SeqTrans is a Transformer based seq2seq neural network with attention and copy mechanisms that aims to fix Java vulnerabilities.
Similar to VRepair, they also first train on a bug fix corpus, and then fine-tune on a vulnerability fix dataset. Their input representation is the vulnerable statement, and the statements that defined the variables used in the vulnerable statement. To reduce the vocabulary, the variable names in the buggy and fixed methods are renamed and they use BPE to further tokenize the tokens. VRepair is different from SeqTrans in that we target fixing C vulnerabilities instead of Java vulnerabilities. Importantly, we utilize the copy mechanism to deal with tokens outside the training vocabulary and not BPE. Our evaluation is done based on two independent vulnerability fix datasets to increase the validity. VRepair's code representation is also different, and allows us to represent multi-line fixes in a compact way.
The patches typically shield the function from malicious input, such as a simple filter rule that blocks certain inputs. ExtractFix is an automated program repair tool that can fix vulnerabilities that can cause a crash  [61]. It works by first capturing constraints on inputs that must be satisfied, the constraints capturing the properties for all possible inputs. Then, they find the candidate fix locations by using the crash location as a starting point, and use control/data dependency analysis to find candidate fix locations. The constraints, together with the candidate fix locations, are used to generate a patch such that the constraints cannot be violated at the crash location in the patched program. MemFix is a static analysis based repair tool for fixing memory deallocation errors in C programs  [62] , e.
g. , memory leak, double free, and use-after-free errors. It does so by first generating candidate patches for each allocated object using a static analyzer. The correct patches are the candidate patches that correctly deallocate all object states. It works by reducing the problem into an exact cover problem and using an SAT solver to find the solution. LeakFix is an automatic program repair tool for fixing memory leaks in C programs  [63]. It first builds the control flow graph of the program and uses intra-procedural analysis to detect and fix memory leaks. The generated fix is checked against a set of procedures that ensures the patch does not interrupt the normal execution of the program. ClearView is an early work that can protect deployed software by automatically patching vulnerabilities  [64].
To do so, ClearView first observes the behavior of software during normal execution and learns invariants that are always satisfied. It uses the learned invariant to detect and repair failures. The patch can change register values and memory locations, or change the control flow. It has been able to resist 10 attacks from an external Red Team. IntRepair is a repair tool that can detect, repair, and validate patches of integer overflows  [65]. It uses symbolic execution to detect integer overflows by comparing the execution graphs with three preconditions. Once an integer overflow is detected, a repair pattern is selected and applied. The resulting patched program is executed again with symbolic execution to check if the integer overflow is repaired. SoupInt is a system for diagnosing and patching integer overflow exploits  [66].
Vulnerability Datasets Big-Vul is a C/C++ code vulnerability dataset collected from open sourced GitHub projects  [15]. It contains 3754 vulnerabilities with 91 different vulnerability types extracted from 348 GitHub projects. Each vulnerability has a list of attributes, including the CVE ID, CWE ID, commit ID, etc. This dataset can be used for vulnerability detection, vulnerability fixing, and analyzing vulnerabilities. CVEfixes  [16]   [70]. They used the dataset to train a bi-directional graph neural network for a vulnerability detection system.