dataesr/scientific-paragraphs-categorization huggingface dataset:
This procedure yielded two resolution maps, for the resolution in horizontal and vertical direction. 2. 2 Determining g-factor maps with a pseudo multiple replica method Model and data driven reconstructions are typically not shift-invariant. The spatially resolved g-factor quantifies any transformation-immanent noise amplification in addition to the reduced SNR due to reduced scan time. In the following, for determination of g-factor maps we use a pseudo multiple replica method similar to  [10]. For some given k-space data, pseudo repetitions of a measurement are simulated by adding synthetic noise. Phased coil arrays exhibit white Gaussian noise, which is correlated due to the coupling of the coils between each other. We thus determined the amplitude and correlation of the noise in the receiver array with a separate acquisition, where no RFpulses are played out.
For gaining insights into their reconstruction properties, we chose a selection of popular model-based and datadriven methods: GRAPPA: As classical parallel imaging baseline, we used a GRAPPA  [11]  implementation  [12]. Compressed sensing: A TV-regularized compressed sensing (CS) model, which is implemented in the Berkley Advanced Reconstruction Toolbox (BART)  [13]. The optimization problem is solved with the ADMM optimizer  [14]. The parameters (number of iterations and regularization weight) were individually adjusted for the T 1 -and T 2weighted image. Variational network: For data-driven reconstructions, we used the publicly available challenge dataset and the baseline models of the 2020 fastMRI challenge  [15]. The pre-trained models were provided by fastMRI. They are trained on the fastMRI 2020 multicoil brain challenge dataset, consisting of 4469 T 1 , T 1 post-contrast, T 2 , and FLAIR acquisitions with additional 1378 for validation.
The dataset and the models are described in detail in the accompanying paper  [15]  and a previous publication  [16]  All methods were applied to data, which were retrospectively undersampled with an acceleration factor of R ¼ 4. Undersampling was applied in phase-encoding direction, which corresponds to horizontal direction in image space. The undersampling masks, which differ between the methods due to their specific requirements, can be seen in Fig. 2. For all methods, a fully sampled region in the center of k-space with size 8% of phase encoding lines, called the autocalibration signal (ACS) region, was used. For GRAPPA, the calibration was performed on this region and the reconstruction then applied to k-space where every fourth line was sampled, without ACS region, to ensure R ¼ 4. For CS, a random undersampling pattern with ACS region was used.
Acquisitions Raw data of a fully sampled MR-investigation of the brain on a clinical 3T scanner (MAGNETOM Prisma fit, Siemens Healthcare GmbH, Erlangen, Germany) was exported as a representative dataset. The subject was a 57year old woman. Images were acquired to rule out cerebral manifestation of a B-Non-Hodgkin Lymphoma. Apart from a few unspecific white matter lesions no intracranial pathology was found. Transversal T 1 -weighted images were acquired using a spoiled gradient echo sequence A head coil with 20 channels was used. As described in section 2. 2, we additionally acquired a noise measurement for computing g-factors. Our study was approved by the institutional review board of the University of Würzburg and the requirement of written informed consent by the study participant was waived due to the retrospective study design.
For datadriven methods, a good match between the training dataset and test images is also important. Investigating these relationships in detail in a more comprehensive study including a variety of anatomies, contrasts, resolutions and hardware would be a worthwhile goal for future work. We found that there is a substantial difference in the width of the LPSF's main lobe for the vertical and horizontal directions. The values for w in the horizontal direction, where undersampling was performed, were clearly higher than in the vertical direction, see Figs. 6 7 8. This might be taken into account when choosing the direction in which to perform undersampling. The general idea of introducing some kind of perturbation to evaluate how a reconstruction method responds has already been used in medical imaging  [4] [5] [6] [7].
GRAPPA interpolation is achieved by a convolution in k-space, separately on the data of each receiver coil. The uniform operation in k-space allows GRAPPA to retain ideal resolution, but at the cost of increased noise. Due to their similarity with GRAPPA in terms of mechanisms and qualitative output, we believe that an assessment of resolution of other parallel imaging techniques like SENSE would have similar results, though a detailed investigation would be of interest. The other examined methods (CS, U-Net and VN) are applied in image space. U-Net operates only on a coil-combined reconstruction. CS and VN use the spatial information by the coil sensitivity maps in their data consistency terms, but not in their regularization terms, which operate on coil-combined images. We consider these mechanisms as a trade-off between resolution and noise, as it allows them to achieve noise suppression and good artifact correction, but results in the introduction of local blurring.
3 observed hallucinations. We want to stress the fact that the dataset was by no means specifically selected for this reason. Other studies have reported similar errors  [15, 17]. Though this is certainly a rare occurrence for well-designed methods, the mere possibility raises the still open question of how these kind of reconstruction faults could be discovered, should they appear in potential clinical application. The methods we presented here can give a spatially resolved indication of the ability to depict small details and the level of noise. High resolution and low noise are certainly important prerequisites for proper reconstructions, and thus the maps we presented can identify areas in which lower quality is possible. However, locally quantifying fidelity of reconstructions to the measured data in a sufficient manner is a further step, and the results presented here do not allow conclusions in this regard.
This is manageable for fast reconstruction methods, but problematic for slower ones. Thus, to overcome this issue, the reconstruction method itself can be accelerated through efficient implementations or appropriate hardware, like GPUs. Still, the determination of local resolution may not be suited for routine computation in many images. A clear advantage of the resolution measurement on the other hand is its simplicity. The algorithm requires in essence only the computation of two reconstructions together with some basic signal processing for determining the width of the main lobe of the LPSF. We provide an implementation on GitHub (https://github. com/expRad/).
The variational network (VN) model consists of 12 cascades, and the regularization terms are realized by U-Nets with 4 up-and downsampling steps and 18 channels in the first layer. The entire VN model has 2:99 Á 10 7 trainable parameters. U-Net: Like the VN model, the U-Net model was one of the baseline models of the 2020 fastMRI challenge [15], and was trained on the same dataset. It has the same structure as the regularizers in the VN model, but uses 256 channels in the first layer. It has a total of 4:96 Á 10 8 trainable parameters.
Â 235 mm [2], in-plane spatial resolution = 0. 61 Â 0. 68 mm 2 , matrix size 384 Â 346) and T 2weighted images were acquired using a TSE-sequence (TE = 91 ms, TR = 6610 ms, flip angle 150°, FOV = 235 Â 235 mm 2 , in-plane spatial resolution = 0. 46 Â 0. 51 mm 2 , matrix size 512 Â 461), well within the distribution of the NYU training dataset.
In this paper, we address the problem of automatic repair of software vulnerabilities with deep learning. The major problem with data-driven vulnerability repair is that the few existing datasets of known confirmed vulnerabilities consist of only a few thousand examples. However, training a deep learning model often requires hundreds of thousands of examples. In this work, we leverage the intuition that the bug fixing task and the vulnerability fixing task are related and that the knowledge learned from bug fixes can be transferred to fixing vulnerabilities. In the machine learning community, this technique is called transfer learning. In this paper, we propose an approach for repairing security vulnerabilities named VRepair which is based on transfer learning. VRepair is first trained on a large bug fix corpus and is then tuned on a vulnerability fix dataset, which is an order of magnitude smaller.
One of the reasons is that deep learning models depend on acquiring a massive amount of training data  [8] , while the amount of confirmed and curated vulnerabilities remains small. Consider the recent related work Vurle  [9] , where the model is trained and evaluated on a dataset of 279 manually identified vulnerabilities. On the other hand, training neural models for a translation task (English to French) has been done using over 41 million sentence pairs  [10]. Another example is the popular summarization dataset CNN-DM  [11]  that contains 300 thousand text pairs. Li et al. showed that the knowledge learned from a small dataset is unreliable and imprecise  [12]. Schmidt et al. found that the error of a model predicting the thermody-Manuscript submitted 2021-04-16. namic stability of solids decreases with the size of training data  [13].
We argue that learning from a small dataset of vulnerabilities suffers from the same problems (and will provide empirical evidence later). In this paper, we address the problem that vulnerability fix datasets are too small to be meaningfully used in a deeplearning model. Our key intuition to mitigate the problem is to use transfer learning. Transfer learning is a technique to transfer knowledge learned from one domain to solve problems in related domains, and it is often used to mitigate the problem of small datasets  [14]. We leverage the similarity of two related software development tasks: bug fixing and vulnerability fixing. In this context, transfer learning means acquiring generic knowledge from a large bug fixing dataset and then transferring the learned knowledge from the bug fixing task to the vulnerability fixing task by tuning it on a smaller vulnerability fixing dataset.
We realize this vision in a novel system for automatically repairing C vulnerabilities called VRepair. To train VRepair, we create a sizeable bug fixing dataset, large enough to be amenable to deep learning. We create this dataset by collecting and analyzing all 892 million GitHub events that happened between 2017-01-01 and 2018-12-31 and using a heuristic technique to extract all bug fix commits. In this way, we obtain a novel dataset consisting of over 21 million bug fixing commits in C code. We use this data to first train VRepair on the task of bug fixing. Next, we use two datasets of vulnerability fixes from previous research, called Big-Vul  [15]  and CVEfixes  [16]. We tune VRepair on the vulnerability fixing task based on both datasets. Our experimental results show that the bug fixing task can be used to train a model meant for vulnerability fixing; the model trained only on the collected bug fix corpus achieves 18.
24% accuracy on Big-Vul and 15. 98% on CVEfixes, which validates our initial intuition that these tasks are profoundly arXiv:2104. 08308v3 [cs. SE] 4 Jan 2022 similar. Next, we show that by using transfer learning, i. e. , by first training on the bug fix corpus and then by tuning the model on the small vulnerability fix dataset, VRepair increases its accuracy to 21. 86% on Big-Vul and 22. 73% on CVEfixes, demonstrating the power of transfer learning. Additionally, we compare transfer learning with a denoising pre-training followed by fine-tuning on the vulnerability fixing task and show that VRepair's process is better than pre-training with a generic denoising task. We also show that transfer learning improves the stability of the final model. In summary, our contributions are: • We introduce VRepair, a Transformer Neural Network Model which targets the problem of vulnerability repair.
The core novelty of VRepair is to employ transfer learning as follows: it is first trained on a bug fix corpus and then tuned on a vulnerability fix dataset. • We empirically demonstrate that on the vulnerability fixing task, the transfer learning VRepair model performs better than the alternatives: 1) VRepair is better than a model trained only on the small vulnerability fix dataset; 2) VRepair is better than a model trained on a large generic bug fix corpus; 3) VRepair is better than a model pre-trained with a denoising task. In addition, we present evidence that the performance of the model trained with transfer learning is stable. • We share all our code and data for facilitating replication and fostering future research on this topic.
Software Vulnerabilities A software vulnerability is a weakness in code that can be exploited by an attacker to perform unauthorized actions. For example, one common kind of vulnerability is a buffer overflow, which allows an attacker to overwrite a buffer's boundary and inject malicious code. Another example is an SQL injection, where malicious SQL statements are inserted into executable queries. The exploitation of vulnerabilities contributes to the hundreds of billions of dollars that cybercrime costs the world economy each year  [17]. Each month, thousands of such vulnerabilities are reported to the Common Vulnerabilities and Exposures (CVE) database. Each one of them is assigned a unique identifier. Each vulnerability with a CVE ID is also assigned to a Common Weakness Enumeration (CWE) category representing the generic type of problem.
Definition a CVE ID identifies a vulnerability within the Common Vulnerabilities and Exposures database. It is a unique alphanumeric assigned to a specific vulnerability. For instance, the entry identified as CVE-2019-9208 is a vulnerability in Wireshark due to a null pointer exception. 2019 is the year in which the CVE ID was assigned or the vulnerability was made public; 9208 uniquely identifies the vulnerability within the year. Definition a CWE ID is a Common Weakness Enumeration and identifies the category that a CVE ID is a part of. CWE categories represent general software weaknesses. For example CVE-2019-9208 is categorized into CWE-476, which is the 'NULL Pointer Dereference' category. In 2020, CWE-79 (Cross-site Scripting), CWE-787 (Out-ofbounds Write) and CWE-20 (Improper Input Validation) are the 3 most common vulnerability categories 1.
Transformers Sequence-to-sequence (seq2seq) learning is a modern machine learning framework that is used to learn mappings between two sequences, typically of words  [18]. It is widely used in automated translation  [19] , text summarization  [20]  and other tasks related to natural language. A seq2seq model consists of two parts, an encoder and a decoder. The encoder maps the input sequence X = (x 0 , x 1 ,. , x n ) to an intermediate continuous representation H = (h 0 , h 1 ,. , h n ). Then, given H, the decoder generates the output sequence Y = (y 0 , y 1 ,. , y m ). Note that the size of the input and output sequences, n and m, can be different. A seq2seq model is optimized on a training dataset to maximize the conditional probability of p(Y | X), which is equivalent to: p(Y | X) = p(y 0 , y 1 ,.
Transfer Learning Traditional machine learning approaches learn from examples in one domain and solve the problem in the same domain (e. g. , learning to recognize cats from a database of cat images). However, as humans, we do better: we are able to apply existing knowledge from other relevant domains to solve new tasks. The approach works well when two tasks share some commonalities, and we are able to solve the new problem faster by starting at a point using previously learned insights. Transfer learning is the concept of transferring knowledge learned in one source task to improve learning in a related target task  [24]. The former is called the source domain and the latter the target domain. Transfer learning is commonly used to mitigate the problem of insufficient training data  [14].
Overview VRepair is a neural model to fix software vulnerabilities, based on the state-of-the-art Transformer architecture. The prototype implementation targets C code and is able to repair intraprocedural vulnerabilities in C functions. VRepair uses transfer learning (see subsection 2. 3) and thus is composed of three phases: source domain training, target domain training, and inference, as shown in Figure  1. Source domain training is our first training phase. We train VRepair using a bug fix corpus because it is relatively easy to collect very large numbers of bug fixes by collecting commits (e. g. , on GitHub). While this corpus is not specific to vulnerabilities, per the vision of transfer learning, VRepair will be able to build some knowledge that would turn valuable for fixing vulnerabilities.
From a technical perspective, training on this corpus sets the neural network weights of VRepair to some reasonable values with respect to manipulating code and generating source code patches in the considered programming language. Target domain training is the second phase after the source domain training. In this second phase, we used a high quality dataset of vulnerability fixes. While this dataset only contains vulnerability fixes, its main limitation is its size because vulnerability fixes are notoriously scarce. Based on this dataset, we further tune the weights in the neural network of VRepair. In this phase, VRepair transfers the knowledge learned from fixing bugs to fixing vulnerabilities. As we will demonstrate later in subsection 5. 2, VRepair performs better with transfer learning than with just training on the small vulnerability fixes dataset.
Inference is the final phase, where VRepair predicts vulnerability fixes on previously unseen functions known to be vulnerable according to a given vulnerability localization technique. VRepair is agnostic with respect to vulnerability localization, it could be for example a human security expert or a static analyzer dedicated to some vulnerability types. For each potentially vulnerable location, VRepair may output multiple predictions which may be transformable into multiple source code patches. Those tentative patches are meant to be verified in order to check whether the vulnerability has disappeared. This means that, like localization, verification is independent of VRepair. It would typically be another human security expert, or the same static analyzer used before.
Code Representation Early works on neural repair  [26]  simply output the full function for the proposed patched code, but the performance of such a system was seen to diminish as the function length increased  [26]. Recently, other output representations have been proposed, including outputting a single line  [22] , and outputting transformations as repair instructions  [27] -  [29]. Inspired by this recent related work, we develop a token-based change description for repairs. It allows for sequences shorter than a full function to be generated, it is easily applied to code, and it can represent any token change necessary for code repair. To sum up, in VRepair, the core idea is to represent the patch as an edit script at the token level. The full overview of the input and output formats for VRepair are shown in Figure  2.
In VRepair, the core idea is that the network only outputs the changed source code tokens, and not the whole function, i. e. , not the whole token sequence. By doing this: 1) the representation allows for representing multiple changes to a function 2) the representation decreases the size of the output sequence to be generated. As seen in box (d) on Figure  2 , our system uses a special token to identify a change, <ModStart>, followed by n context tokens which locate where the change should start by specifying the tokens before a change. <ModEnd> is used when tokens from the input program are being replaced or deleted, and it is followed by n context tokens to specify the completion of a change. Definition a token context diff is a full change description for a function, identifying multiple change locations on multiple lines, using two special tokens <ModStart> and <ModEnd> to identify the start and end contexts for change locations.
But as shown in box (g), the context size 2 specification has multiple interpretations for the token replacement change. This is because '{ return' and '; }' both occur 2 times in the program resulting in 3 possible ways the output specification can be interpreted. For example, the first time the tokens '{ return' occur in the buggy source code is directly before 'arrax[index]', which is not the correct location to begin the modification to produce the correct repaired function. In this example, we see that a 2-token context is ambiguous whereas 3 tokens are sufficient to uniquely identify a single possible patch. Our pilot experiments showed that a context size of 3 successfully represents most patches without ambiguity, and hence we use this context size to further develop VRepair. In addition to confirming a context size of 3, our initial experiments have shown that the token context diff approach supports multi-line fixes, which is an improvement on prior work only attempting to repair single lines  [22].
We note that 49% of our source domain dataset which we will use in the experiment (see subsection 4. 2) contains multi-line fixes, demonstrating the importance of the token context diff. We also present in subsection 5. 3 a case study of such a multiline patch. In summary, VRepair introduces a novel context-based representation for code patches, usable by a neural network to predict the location where specific tokens need to be added, deleted, or modified. Compared to other works which limit changes to single lines  [22]  or try to output the entire code of repaired functions  [26] ,  [35] , our approach allows for complex multi-line changes to be specified with a small number of output tokens.
Tokenization In this work we use a programming language tokenizer with no subtokenization. We use Clang as the tokenizer because it is the most powerful tokenizer we are aware of, able to tokenize un-preprocessed C source code. We use the copy mechanism to deal with the out-of-vocabulary problem per  [22]. We do not use sub-tokenization such as BPE  [36]  because it increases the input and output length too much (per the literature  [34] , confirmed in our pilot experiments). Variable renaming is a technique that renames function names, integers, string literals, etc. to a pool of predefined tokens. For example function names GetResult and UpdateCounter can be replaced with FUNC_1 and FUNC_2. We do not use variable renaming because it hides valuable information about the variable that can be learned by word embeddings.
For the bug fix corpus where we don't have this information, we use the 'CWE-000' token meaning "generic fix". This special token is mandatory for target domain training and inference as well, as we shall see later. In machine learning, overfitting can occur when a model has begun to learn the training dataset too well and does not generalize well to unseen data samples. To combat this issue we use the common practice of early stopping  [37]  during source domain training. For early stopping, a subset of our generic bug fix dataset is withheld for model validation during training. If the validation accuracy does not improve after two evaluations, we stop the source domain training phase and use the model with the highest validation accuracy for target domain training.