VQR: Automated Software Vulnerability Repair Through Vulnerability Queries
Authors: Michael Fu
Van Nguyen
Chakkrit Tantithamthavorn
Trung Le
Dinh Phung
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automated Vulnerability Repair
Cross-Attention Mechanism
Transformers-based Models

Abstract: Recently, automated vulnerability repair (AVR) approaches have been widely adopted to combat the increasing number of software security issues. In particular, transformer-based models achieve competitive results. While existing models are learned to generate vulnerability repairs, existing AVR models lack a mechanism to provide their models with the precise location of vulnerable code (i.e., models may generate repairs for the non-vulnerable areas). To address this problem, we base our framework on the VIT-based approaches for object detection that learn to locate bounding boxes via the cross-matching between object queries and image patches. We cross-match vulnerability queries and their corresponding vulnerable code areas through the cross-attention mechanism to generate more accurate repairs. To strengthen our cross-matching, we propose to learn a novel vulnerability query mask that greatly focuses on vulnerable code areas and integrate it into the cross-attention. Moreover, we also incorporate the vulnerability query mask into the self-attention to learn embeddings that emphasize the vulnerable areas of a program. Through an extensive evaluation using the real-world 5,417 vulnerabilities, our approach outperforms all of the baseline methods by 3.39%-33.21%. The training code and pre-trained models are available at https://github.com/AVR-VQR/VQR.

Under review as a conference paper at ICLR 2023
VQR: AUTOMATED SOFTWARE VULNERABILITY RE- PAIR THROUGH VULNERABILITY QUERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, automated vuinerability repair (AVR) approaches have been widely adopted to combat the increasing number software security issues. In particular, transformer-bast based models achieve competitive results. While existing models are learned to generate vulnerability repairs, existing AVR models lack mechanism provide their models with the precise location of vulnerable code (i.e., models may generate repairs for the non-vulnerable areas). To address this problem, we base our framework detection that learn to locate bounding boxes via the cross-matching between ob- ject queries and image patches. We cross-match vulnerability queries and their corresponding vulnerable code areas through the cross- cross-attention mechanism to generate more accurate repairs. To strengthen our cross- matching,,, we propose to learn novel vulnerability query mask that greatly focuses on vulnerable code areas and integrate into the cross-attention. Moreover, we also incorporate the vulnerability query mask into the self-attention to learn embeddings that empha- size the vulnerable areas of a program. Through an extensive evaluation using approach outperforms all of the baseline
the vision transformer(VTT--boan
approaches for object
the real-world 5, 5,417 vulnerabilities, methods by 2.68%-3..33%. The training code and pre-trained models are avail- able at /tttts::/ggtthaiee com AVR- VOR/VOR
INTRODUCTION
Software vulnerabilities are security flaws, glitches, or weaknesses found in software code that could lead to severe system crash be leveraged as a threat source by attackers (CSRC 2020). Accord- ing to National Vulnerability Database (NVD), the number of vulnerabilities discovered yearly has increased from 6.447 2016 to 20.156 in 2021 and 18,017 vulnerabilities have been found in 2022. This trend indicates more vulnerabilities are being discovered and released every year, meaning that there will more workloads for security analysts to track down and patch those vulnerabilities. In particular, it may take 58 days average to fix vulnerability based vulnerability statistics reported in 2022 (Edgescan) (022). Recently, Deep Learning (DL)-based approaches have been pro- posed to automate the vulnerability repair process by learning the representation of vulnerable pro- grams and generating repair patches accordingly, which may potentially accelerate manual security analysis processes. Specifically, the transformer architecture has been widely adopted to generate accurate vulnerability patches that repair the vulnerable code automatically (Chen et al. [2022] Chi al. 2022, Berabi et al. 202T; Fu et al. 2022). The attention-based transformer is shown to be more effective than RNNs because its self-attention mechanism learns global dependencies when scanning through each word embedding rather than processing input sequentially.
For the software vulnerability repair (SVR) problem, awareness and attention to the vulnerable code areas including vulnerable statements are crucially important. This further helps to guide an SVR model to emphasize and focus more on the vulnerable statements for producing better repairs. However, it is challenging because the vulnerable areas locate spatially a source code. Toward this challenge, we observe that object detection in computer vision intuitively shares a similar concept to vulnerability repair because both approaches need to localize specific items in the input. Particularly, by linking the vulnerable code areas in souree code to the objects in image, we hope to borrow the principles from the VIT-based objection detection approaches (Carion et al. 2020 Zhu et al. 2020 Wang et al. 2021a) propose novel solution for the SVR problem.

---

Under review as a conference paper at ICLR 2023
Object Detection
Vumerabiiity Localizatin
Sourcn Code
Object Quenka
Wanerabiltty Quaries
Inherablee
Code Block
Use atgoct quemtyy tocete bouadding boxere
Use vudnerabilty spartes to locete vudromatile oode
Figure I: Intuitively, not all code tokens in a program need to be repaired and the repair can be in multiple areas. Similarly, not all pixels in an image has objects and the objects can appear in multiple locations in an image. Thus, in object detection, object queries are used in VIT-based approaches (Carion et al.) 2020; Zhu al. 2020 Wang al., 202 to predict bounding boxes and locate objects. With similar principle of object detection, we leverage vulnerability queries to attend more to the vulnerable code tokens in the vulnerable code areas and generate repairs for them.
Specifically, approach is inspired by the VIT-based approaches for object detection (Carion iet al., [2020] Zhu et al. 2020; Wang et al., 2021a) where we connect detecting spatial objects in an image for predicting bounding boxes to localizing vulnerable code tokens in source code for generating the repair tokens. Our model consists of vulnerability repair encoder to produce code token embeddings for code tokens and vulnerability repair decoder to generate repair tokens. Similar to the object queries in the VIT-based approaches for object detection aiming to attend to objects an image for predicting the corresponding bounding boxes, we devise vulnerability queries (VQ) aiming to attend to the vulnerable areas in source code for predicting repair tokens. Additionally, the cross-attention mechanism employed the vulnerability repair decoder assists the VQs in cross-matching and paying more attention to the vulnerable code areas.
Furthermore, for world software vulnerabilities, not all code tokens source code are con- sidered vulnerable, meaning that only some of the code tokens likely to be more vulnerable than the others (Nguyen et al. [202]] Fu & Tantithamthavom, (002). To strengthen the attention of the VQs the vulnerable code areas code tokens, we train additional model to learn vul- nerability mask. Specifically, given a source code, the vulnerability mask has significantly higher vulnerability scores for the vulnerable code tokens. We then apply the vulnerability mask both the vulnerability repair encoder/decoder to enrich our approach, named Vulnerability Query based Software Vulnerability Repair (VQR). Finally, Figure presents conceptual overview of our VQR.
In summary, our contributions are (i) novel vulnerability repair framework based on object detec- tion that uses vulnerability queries to generate repair patches; (ii) novel vulnerability query mask that facilitates the repair model to locate vulnerable code tokens more accurately during vulnera- bility query: (iii) a comprehensive evaluation of our proposed approach against other automated vulnerability repair approaches using a benchmark dataset including real-world vulnerabilities.
RELATED WORK
Automated Vulnerability Repair (AVR) is task that uses machine learning models to gener- ate repair patches for vulnerable programs. RNN-based models such as SequenceR (Chen et al.] 2019) have been proposed encode the vulnerable programs and decode corresponding repairs quentially. SequenceR used Bi-LSTMs as encoders with unidirectional LSTMs to generate repairs. Recently, attention-bau based Transformer models have been leveraged in the AVR domain, which was shown to be more accurate than RNNs. For instance, VRepair (Chen et al. 2022) relied on an encoder-decoder Transformer with transfer learning using the bug-fix data boost the performance of the vulnerability repair on C/C++ programs. Seq'Trans (Chi et al.) (022) constructed code se- quences by considering data flow dependencies of programs and leveraged an identical architecture as VRepair. On the other hand, Berabi et al. (2021) proposed use T5 model pre-trained on natural language corpus (i.c., T5-large (Raffel et al. 20200) to fix JavaScript programs and Fu et al.) 2022) utilized T5 model pre-trained on source code (i.e., CodeT5 (Wang et al.) 2021b)) pair C/C++ programs. Additionally, MMashaddi & Hemmati (202T) applied the CodeBERT (Feng al. 2020 model repair Java bugs. Those large pre-trained language models have demonstrated
2

---

Under review as a conference paper at ICLR 2023
strong improvement over RNNs and non non-pretrained transformers because the pre-training steps help the models gain better initial weights for the vulnerability repair downstream task than training from scratch. DLFix (Li et al. 2020) and CURE (Jiang et al.) 202T were proposed to generate vulner- ability repairs that satisfy test cases. Thus, complete repaired functions are required to train and evaluate models and the problem statement is different from ours described in Section 3.1 Differ- ent from the sequence-based methods mentioned above, Dinella et al. (2020) proposed to learn the graph transformation based on the Abstract Syntax Tree (AST) of source code, which used GNNs to represent the program and LSTMs to generate repairs for JavaScript programs.
Vulnerable Function CWE-787 (Out-af-bounds Write)
Repaired Function
41
41
GPMF ERR 2GPM stream "ma, unnt32_ siba)
41
41
GPMF
ERR SvalldSzo(GFFF staam "ms, uln132_ _ [920)
42
42
42
42
43
43
if [ms]
43
43
if (ms)
44
44
44
44
45
int52_1 I nestsize - (int32 timms-nnest sicp///sssieeeeeeeeeeee
45
uint32 It nestsize (uint32_tims--ness fims-/mmmt sicefmp--nestt Jevel):
46
48
(nesssiioe ma-onest_ Jevel 0)
46
48
(nealsine && ma-onest Jevel s= D)
nestsize ms- butter size longs;
47
47
nestsize ms-nbutfor size longs;
50
GPMF ERROR_ BAD STRUCTURE
51
GPMF ERROR BAD STRUCTURE
52
Subword-tooknt the vulnerable function
Subword vulnerability
[5,,.... -n]
[GP MP: 'ERR', "laValdd, 'Sizel, 'GP. IMF: 'stream" Imme luint: 32 Y, Sizel, Y. T. 'P. (,. Imst '/, int, (32) 'nest, litel.
[mma: T. 'uinet, -! Y. "reetsize'. V. 'uine', Y. 7. mea '>]
=, Y, Inf, 121 ". 'ms, lo), hest, 'size' TI: "ms, 'nest, lever, It If, heett 'size" feet, V. '88; 'met, hat, hneat" lleell leel, '..
T. heat" lazel lel 'mas tat, 'buffar," 'iize" ": longs V. 1f, 'ibe'.
4 cee, hest, "azee V, 'reuum, IGP, MMF T. 1KK:
freuurn, IGP..
MP, ERROR. BADY 'STRUCT, URE, 1,
Figure 2: (CWE-787 Out-of-bounds Write) A real-world example (GoPro 2019) of vulnerability in function caused by an inappropriate variable type definition, which could lead to serious security breaches or system crashes. The left column presents the vulnerable function where below are sub-word tokens xy used as input for our repair model. It can be seen that only some of the tokens highlighted in red (i.e., tokens corresponding to Line 45) are vulnerable. The right column presents the corresponding repaired function where below are sub-word tokens ys as the repair patch output by our repair model.
3 OUR PROPOSED APPROACH
3.1 PROBLEM STATEMENT
Assuming have source code data set consisting of vulnerable source code functions along with corresponding repair patches that repair the vulnerable parts of those functions. We denote the data set D { (I), y1,,... (IN.yN)), }, where 2; is a vulnerable function and y, is its repair patch. Note that each yı is not a complete function but a patch used to repair the vulnerable part in the corresponding 24 as shown in Figure 2 The mapping between 2; and yi has been completed by [Chen et al.] (2022) through parsing the code difference between the vulnerable and the fixed version of the source functions. In this paper, we leverage BPE algorithm (Sennrich et al. 2016) to tokenize and consider z; sequence of code tokens denoted as 2; = [t,, t2, ..., t,, where the code token ] = 1,.... n could be clean token or vulnerable token (i.e.. the tokens highlighted in red in Figure 2b. Similarly, repair patch y/ [ri, ..., r'] where yı consists of k number of repair tokens j 1, 1..,, k. Each code token L, and repair token ryy will be embedded into a vector for the model to learn its representation as detailed in Section 3.2) We define this problem as sequence-to-
sequence code generation task with an objective to capture vulnerable code tokens in x; to generate corresponding repair patch y:-
3.2 VULNERABILITY QUERY BASED SOFTWARE VULNERABILITY REPAIR
Our approach is inspired by the VIT-based approaches (Carion et al. 2020 Zhu et al. 2020 Wang et al. 2021a) for object detection where we link detecting spatial objects in image for predicting
3

---

Under review as a conference paper at ICLR 2023
Vulnerability Repair Decoders
Mix
Linor Layer
Vulnerability Repair Encoders
Vulnerability Query Mask
M(x))
Input
V22
V0:
f
with Vuinerabiliiy Mask
Figure 3: An overview architecture of our VQR approach. Input tokens 2; [t...... fx] and vulnerability query mask (x:)) are input to repair encoders that output embeddings input tokens H wherein (x;)) helps to emphasize the vulnerable embeddings. In repair decoders, each vulnerability query VQ; is initialized from the previous repair token r:-1, which is forwarded through multiple decoder layers followed by linear layer to generate a repair token rą- In each repair decoder, cross-attention with M (x;) to emphasize vulnerable embeddings is leveraged to cross-match VQ; and H. and generate repairs corresponding to the vulnerable tokens.
bounding boxes to localizing vulnerable code tokens a source code for generating the repair tokens. Our model consists of encoder produce code token embeddings for code tokens and decoder to generate repair tokens.
Both encoder and decoder are developed based the transformer architecture (Vaswani et al. 2017]. The main component of the encoder is multi-head self-attentions with the aim to learn code token embeddings. Similar DeTR (Carion al. 22200). the decoder utilizes both multi-head self- attentions and cross-attentions, The purpose of the cross-attentions is to cross-match vulnerability queries and their corresponding vulnerable code tokens in vulnerable area. Ideally, when vul- nerability queries achieve good matches with their vulnerable code tokens. they possess sufficient information to generate repair tokens.
Additionally, to orient the matching process for attending more to vulnerable code tokens inside a source code, we propose learn a vulnerability query mask and apply self-attention and decoder cross-attention mechanism. Particularly, vulnerable tokens to train an additional model that outputs the possibility of a code token being vulnerable token. We then base these vulnerable scores conduct vulnerability query mask.
both the encoder rely the information of
In what follows, we present the technicality of the vulnerability repair encoder, the vulnerability repair decoder, and how to conduct and incorporate vulnerability query masks into framework.
3.2. VULNERABILITY REPAIR ENCODER
The purpose of the encoder is to produce code token embeddings for given source code. Each token embedded into a vector in 4d7766 by an embedding layer and input to the first encoder block. A stack of encoder blocks is leveraged encode the representation for embedded sequence through their self-attention layers followed by feed-forward neural networks, and each encoder block can be described as follows:
A' = LN (Mult + H
Enc LN(FFN(A') - A
where the hidden states from the previous encoder block H'-1 forwards through a multi-hent Meali-hhadd self- attention M Atin followed by 2-layer feed-forward neural network FF N, and layer normal-
4

---

Under review as a conference paper at ICLR 2023
ization LN. The process will iterate until we obtain the last encoder hidden states H represent the vulnerable function. Here we note that is the number of encoder blocks applied and H contains the code token embeddings.
3.2.2 VULNERABILITY REPAIR DECODER
Input the vulnerability repair decoder is the vulnerability queries (VQ), each of which aims to match and capture information of vulnerable code tokens in a given source code.
The first VQ embeddings Q = ...; blocks. In each block, we apply both multi-head self-attention and cross-attention as follows:
...,.... L. conducted and fed through several following decoder
Q' LN (N(MMlli.Attt(((")) + Q'
A cross IN(Cross (T + Q'
Q' LN(FFN(AArre)) + Across)
where H is the encoder output.
is worth noting that the cross-attention CrossAthr assists us cross-matching the vulnerability query embeddings Q' = Iq..... of and the code token embeddings. If trained appropriately, the vulnerability query embeddings % % attend and emphasize more the vulnerable code token embeddings in the vulnerable function, which finally contain sufficient information to generate the repair tokens.
Eventually, we obtain the output VQ embeddings Q²= - .6. where U is the number of the decoder blocks applied. On top of these VQ embeddings, we predict the repair tokens Specifically, dedicate a linear layer each VQ embedding qu ..... % and aim to predict T1, Th and Tk+1 = ER (i.c., the end repair token) by maximizing the likelihood with respect to mini-batch of z;:
&
plu xx) = p(ri...., fi,..., t,) II p( of))
(I)
==0
where =[t1, [tt,,.., ii.......t, fn] is the source code and y: Im,,..., ... Tkl is the corresponding repair patch.
The next arising question is how to initialize the first VQ embeddings Q = 198 L Differ- from VIT-based object detection approaches (Carion al. 2020 Zhu et al. 2020 Wang al.
2021ah, we do not initialize the first VQ embeddings Q -..... K randomly. ly. Indeed, we ini-
tialize Q0 lg..-..,00 ol more informatively by setting df SR (i.e., the specific embedding for
the start repairing token), of Tj.j = 1,..., k. By this informative initialization, we cast the vulnerability repair problem to the source code to repair-patch generation task.
The inference process is hence very natural. Given a source code 2; (e., (n]. we pass through the vulnerability repair encoder work out the encoder output H. We start with the first VQ embedding ob SR and feed to the vulnerability repair decoder to generate the first repair token r1- We then set VQ embedding () = r11 and feed it to the vulnerability repair decoder to generate the second repair token r2. We repeat this process until reaching the FR token.
As mentioned before, the key factor to the success of our approach how to accurately cross- match between the vulnerability queries and the vulnerable code tokens of a given source code. Currently, we expect that the cross-attention mechanism guided by maximizing the likelihood in Eq. ① supports in realizing this. To further strengthen the cross moosshnath,ng, learn vulnerability query mask that highly focuses on the vulnerable code tokens and then apply it to the encoder self- attention and the decoder cross-attention mechanism.
3.2.3 LEARNING AND INCORPORATING VULNERABILLTY QUERY MASK
In what follows, we present how learn vulnerability query mask and then apply model.
5

---

Under review as a conference paper at ICLR 2023
1000
Figure 4: The plots of the vulnerability mask transformation to see how 3 control the sharpness. It can be seen that controls the sharpness (i.e., how fast the curve gets saturated), while 3 controls the gap between vulnerable and non-vulnerable scores.
Learning vulnerability query mask. We note that for our dataset D (11, (I UN } each vulnerable function 2y [,,..., Ln] is sequence of code token in which we know exactly the vulnerable scope or information if a code token ij belongs to a vulnerable statement. words, also possess the token-level vulnerable label U; [",,.... ..... u,) wherein Uj 1 means that the code token ły belongs to a vulnerable statement and otherwise. For example, in the source code presented in Figure 2 the code tokens highlighted in red are the vulnerable code tokens labelled 1.
other
We now take advantage of this crucial information to learn vulnerability query masks. Basically, we train an additional model to predict the vulnerability query masks. Specifically, we leverage pre- trained CodeBERT (Feng et al. (220)) model in learning the vulnerability query masks. Each fa in a; embedded into a vector in 7os and forwarded through layers of the BERT architecture. We then use global max pooling layer and a sigmoid activation to obtain the probability mask m(x;) and minimize the following cross-entropy loss with respect to mini-batch of x;
n
H(,,, 02)
My log (x:) + (1 u,) log (1 my (x;))
Finally, to sharpen the vulnerability query mask, we apply the following transformation
B
M (2x)
+ exp(-a(m(x.) 0.5)
where a > 0 and B > 0 are two parameters to control the sharpness of the vulnerability query mask.
In Figure 4 we visualize how and / affect the vulnerability query masks. be seen that controls the sharpness (i.e., how fast the curve gets saturated), while 3 controls the gap between vulnerable and non-vulneeeble scores.
Applying vulnerability query mask our model. We incorporate our vulnerability query mask (VQM) into both encoder output and the cross-attention. For the encoder, we apply as follows:
A' = LN (MMlliiiAtinn ( M(x;) s Multi Altmi HH--)) H
Hi LN(FFN(A) A²)
where @ is the element-wiss product which returns (M) with B Multi AulHiiAtt((H...) For the cross-attention in the decoder, we apply as follows:
Q' LN(MulliAun Q
Arroc LN (Crosss.Attn)() H (xx) @ H ()=n))++ +Q--1
Q' LN(FPN + A
Finally, the entire framework of our approach encapsulated the vulnerability repair encoder, vulner- ability repair decoder, and how incorporate vulnerability masks are summarized in Figure

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We compare our proposed method VQR with existing baseline approaches introduced in Ap- pendix A.2
4.1 EXPERIMENTAL DATASET
We use the same experimental dataset provided by Chen et al. (2022) to evaluate our approach. The dataset consists of Big-Vul Vul (Fan al., (020) and CVEfixes (Bhandari et al. 202T vulnerability fix corpus written in C/C++. The Big- Vul dataset was collected from 348 open-source GitHub projects by crawling the Common Vulnerabilities and Exposures (CVE) database. total, Big-Vul contains 3,754 code vulnerabilities from 2002 to 2019. On the other hand, the CVEfixes dataset was constructed similarly to the Big-Vul, which consists of 5,365 vulnerabilities collected from 1,754 projects from 1999 to 2021. Specifically, we leverage both datasets pre- reepproesssed by [Chen jet al. (2022) and obtain 5,417 samples spanning 2,095 different vulnerabilities (i.e., CVE-ID) after dropping null and duplicate samples.
4.2 PARAMETER SETTING
We split the data into 70% for training. 10% for validation, and 20% for testing. We use pre- trained TS model provided by Wang et al. (2021b), which was pre-trained using multiple denoising objectives related to programming languages. Details of the hyperparameter settings for our method during both pre-training and fine-tuning are in Appendix A.1
4.3 MODEL TRAINING
Given that the existing vulncrability repair dataset only contains limited samples, pre-training a larger bug fix dataset can further enhance the performance of a vulnerability repair model as demonstrated by Chen al. (2022). The intuition is that the software vulnerability is sub-domain of the software defect (i.e., bugs) domain which increases the transferability between the two tasks. Thus, for each model including ours, we first pre-train on the bug fix dataset provided by Chen et al. (2022), which consists of 23,607 samples to obtain more meaningful pre- pre-trained weights for the vulnerability repair downstream task. Note that the bug fix dataset is not overlapping with our experimental dataset introduced in Section 4.1 We train our model through specific epochs as reported Appendix A. and select the best model based on the lowest CE loss on the validation set. We run our experiments on Linux machine with an AMD Ryzen 9 5950X, 64 GB of RAM, and an NVIDIA RTX3090 GPU.
4.4
EXPERIMENTAL RESULTS
4.4.1
MAIN RESULTS
We conduct experiment five times by setting different random seeds and using the dataset deseribed in Section 4.1 and compare our proposed method with the baselines introduced in Ap- pendix A.2 We leverage the percentage of perfect prediction (%PP) evaluation measure as used by Chen et al. 22022) and Fu et al. (022)). %PP is computed as the total correct predictions divided by the total testing samples. specifically, prediction is considered correct if any of the beam search outputs is exactly the same as the ground-truth repair. During beam search, we use beam E [1,3, 5] evaluate all of the methods. Such beam settings lead fewer repair candidates generated by the models, which would be more practical in real-world scenarios so developers will not need to inspect many repair candidates. Additionally, we compute BLEU (Papineni et al., (002)) and Me- teor (Banerjee & Lavie (005) score to measure the similarity between predictions and true labels. The experimental results are shown Table [ Our method outperforms all baselines regardless of the number of beams. When comparing only the top-1 repair candidates (i.e., beam = 1), our VQR is 2.68%-16.9244 better than large pre-trained transformer approaches (i.e., VulRepair and TFix) while method is 23.18%%%4.866 better when comparing with BERT-based approaches (i.e., CoRT and GraphCodeBERT), These results confirm that our proposed method cross- matching vulnerability queries with vulnerable code tokens can help the model encode more mean-
7

---

Under review as a conference paper at ICLR 2023
ingful representation for a vulnerable function and decode the corresponding repair more accurately. In addition, we analyze the performance the approach for each CWE-ID in Appendix A.3
Table 1: (Main results) The comparison between our VQR approach and other baselines. Accuracy presented in percentage. Beam=k shows the measure of %PP. The BLEU and Meteor com- puted based on Beam=5.
Methods
Beam=1
Beam=3
Beam=5
BLEU
Meteor
(Qu(O
3.12
42.72:10.86
0.86
59.11
68.45.1.22
VulRepair
29.65++1227
39.85=+1.31
1..7
58.7=1126
68.1+1.4
TFix
15.41=11.96
2..7=1.69
30:1
50.71+2.07
60.04:1..14
GraphCodeBERT
9.15:10.43
16.83+10.85
21.38:10.54
49.38:10,45
64.4:10.66
CodeBERT
7.47:10.61
13.69
16.85.10.17
43.56:0..9
59.23-11.01
VRepair
5.36
10.31
13.12+0.53
40.06=10.62
55.31+0.63
SequenceR
0.010
0.44
0.53+0.27
7.39
27.74#1.6
4.4.2 ABLATION STUDY
(1) Study the effectiveness of our proposed vulnerability query and mask. To this end, we compare proposed method with other variants follows:
Perfect Vulnerability Masking in Encoders and Decoders: This method uses an identical architecture as our VQR, however, the perfect vulnerability masks (i.e., the exact location of each vulnerable token) are provided instead of predicted by localization model.
This method only applies vulnerability query masks on the self-attention output each encoder to help the model focus more on vulner- able tokens when encoding the representations for vulnerable function.
Vulnerability Query Masking in Encoders:
Vulnerability Query Masking in Decoders: This method only applies vulnerability query masks the decoder rross-attention when cross-matching vulnerability queries and vul- nerable code tokens to support the model to focus more on vulnerable tokens when gener- ating repair tokens.
Without Vulnerability Query Masking: This method is plain transformer encoder- decoder architecture that applies no vulnerability query mask.
With Vulnerability Query Randomly Initialized: This method applies vulnerability query masks both encoders and decoders while vulnerability queries are randomly ini- tialized the start of training.
The experimental results are shown in Table can be seen that our approach to initialize the vul- nerability query (VQ) based on repair tokens during training consistently outperforms the randomly initialized VQ. However, the random VQ method still outperforms baselines such as CodeBERT and BraRhCodeBERT, highlighting the effectiveness of using vulnerability queries with cross-attention (as proposed in Section 3.2.2 for our vulnerability repair task. Moreover, applying the vulnerability query mask is beneficial for both encoder self-attention and decoder cross-attention. It enhances the %PP by 2.93% when applied to encoders while gaining a %PP 3.02% when applied to decoders. While the vulnerability query mask benefits both encoders and decoders, our proposed method to leverage the mask on both sides achieves better results for beam E [1,3]. These results confirm that using our mask on both encoders and decoders more beneficial than using it either side. Last but not least, the perfect vulnerability query mask achieves the highest %PP, highlighting the effectiveness of vulnerability query mask.
To further demonstrate that similar to the capability of object queries in capturing information of bounding objects in images, vulnerability queries (VQ) are capable of attending to and capturing information of vulnerable scopes in source codes. We visualize the cross-attention map between VQs (x-axis) and all token representations of source code (y-axis) in Figure 5 As we expect, the VQs attend to vulnerable scopes, hence capturing sufficient information for predicting the repair patches. More visualizations can be found in Appendix A.4
8

---

Under review as a conference paper at ICLR 2023
Table 2: (Ablation results) The comparison between our proposed method and four other variants.
Accuracy is presented in percentage.
Methods
Beam=1
Beam=3
Beam=5
Perfect Mask Encoder Perfect Mask Decoder
33.76
44.31
46.88
Vul Mask Encoder + Vul Mask Decoder (ours)
33.21
44.04
46.06
Vul Mask Encoder
32.75
43.49
46.15
Vul Mask Decoder
32.84
43.85
45.69
w/o Vul Mask
29.82
39.72
43.67
with Vul Query randomly initialized
12.57
24.95
28.81
Vadrenabllyy Qvery (PC)
Iinerrsiiity Osery (10)
Figure 5: The visualization of the cross-attention scores between vulnerability queries (axis X) and vulnerable code representation (axis Y) in two vulnerable functions. The vulnerable scopes are highlighted in red boxes. It can be seen that the attention scores are highly activated between the interaction of vulnerable tokens representation and vulnerability queries.
(2) Study the effectiveness of pre-training on bug fix corpus. We aim study whether pre- training on a larger bug fix corpus support AVR models to perform better vulnerability repairs. To this end, we directly train each baseline ap- proach and our VQR on the vulnerability repair data without pre-training
Table Compare our method with baselines, where all the methods are not pre-trained on the bug-fix data [Chnn et al. (2022). Accuracy is pre- sented in percentage.
Methods
Beam=1
Beam=3
Beam=
VQR
5.32
8.81
9.72
the bug fix data.
VulRepair
4.13
6.06
7.43
The results shown in Table 3 correspond to the finding by Chen al. (2022) that the knowl- edge from the general bug fix corpus can be transferred to benefit the performance of AVR models. Our method still outperforms all base- line approaches.
TFix
2.75
4.4
4.68
GraphCodeBERT
2.57
4.13
5.23
CodeBERT
1.56
2.29
2.75
VRepair
0.09
0.55
0.92
SequenceR
0
0
0
5 CONCLUSION
In this paper, we have introduced a new AVR method inspired by VIT-based approaches for object detection enhance awareness and attention to vulnerable code areas in a vulnerable function for producing better repairs. In our repair model, we cross-match vulnerability queries and their corre- sponding vulnerable code areas and their corresponding repairs via the cross-attention mechanism. propose to learn a vulnerability query mask that highly
To strengthen such cross-matchings, focuses on vulnerable code areas and incorporate into the cross-an attesstitenion. Additionally, we also apply the vulnerability query mask in the self-attention of encoders to help model focus more on vulnerable code tokens when learning the embeddings of each token. Through an extensive eval- uation of 5,417 real-world vulnerabilities, approach outperforms all of the baseline approaches.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. Thi: Learning to fix coding errors with text-to-text transformer. In the Proceedings of the International Conference Machine Learning (ICML), pp. 780-791. PMLR, 2021.
Guru Bhandari, Amara Naseer, and Leon Moonen. Cvefixes: automated collection of vulnerabilities and their fixes from open-source software. In the Proceedings of the 17th International Confer- ence on Predictive Models and Data Analytics Software Engineering, pp. 30-39, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In the Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-229. Springer, 2020.
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-ssquenee learning for end-to-end program repair. IEEE Transactions on Software Engineering (TSE), 47(9):1943-1955 2019.
Zimin Chen, Steve Kommrusch, and Martin Monperrus. Neural transfer learning for repairing
curity vulnerabilities in с code. IEEE Transactions on Software Engineering (TSE), 2022. doi: 10.1 209/0SE.
08308
Jianlei Chi, Yu Qu, Ting Liu, Qinghua Zheng, and Heng Yin. Seqtrans: Automatic vulnerability fix via sequence to sequence learning. IEEE Transactions on Software Engineering (TSE), 2022.
CSRC. Definition of software vulnerability. htt.::://care.nnss..ooo////
software, wereeevuleeeeeee r wulnerability 2020.
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In the Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Edgescan.
2022
vulnerability statistic report.
https://www.eddessca com/
2022.
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. A clc++ code vulnerability dataset with code changes and eve summaries. In the Proceedings of the 17th International Conference on Mining Software Repositories, pp. 508-512, 2020.
Zhangyin Feng, Daya Guo, Duyu Tang. Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In the Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1536-1547, 2020.
Michael Fu and Chakkrit Tantithamthvvom. Linevul: transformer-based line-level vulnerability prediction. In the Proceedings of the 19th International Conference on Mining Software Reposi- tories (MSR), 2022.
Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dhung Dinh. Vulrepair: A (5-based automated software vuinerability repair. In the Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE),
GoPro. tems.
An
example software
vulnerability from gopro
sys-
https://github. com wttpprr/pmmf-ppaeeeee
t./
2019.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Grapheodebert: Pre-training code representations with data flow. In the Proceedings of the International Conference on Learning Representations (ICLR), 2021.
10

---

Under review as a conference paper at ICLR 2023
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint (rXi::1909.00436. 2019.
Nan Jiang, Thibaud Lutellier, and Lin Tan. Cure: Code-aware neural machine translation for auto- matic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineer- ing (ICSE), pp. 1161-1173, IEEE, 2021.
Yi Li, Shaohua Wang, and Tien Nguyen. Difix: Context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd International Conference Software Engineering (ICSE), pp. 602-614, 2020.
Ehsan Mashhadi and Hadi Hemmati. Applying codebert for automated program repair of java simple bugs. In the Proceedings of the International Conference Mining Software Repositories (MSR), pp. 505-509. IEEE, 2021.
Van Nguyen, Trung Le, Olivier de Vel, Paul Montague, John Grundy, and Dinh Phung. Information- theoretic source code vulnerability highlighting. the Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL), pp. 311-318, 2002.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21 21(140):1-67, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In the Proceedings of the Association for Computational Linguistics (ACL), pp-
1715-1725, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In the Proceedings of the Ad- vances in neural information processing systems (NIPS), volume 30, 2017.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-bas object detection. arXiv preprint arXiv:2109.07107, 2021a.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. In the Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8696-8708, 2021b.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In the Proceedings of the International Conference on Learning Representations (ICLR), 2020.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING SCHEME OF OUR VQR APPROACH
Table Training scheme of our VQR approach. Note. #: Scheme for training the mask prediction
model; #: Scheme for training the repair model.
Training
Data
Segene
Seqiee
Optim
Sch.
LR
Grad Clip
Bhz
Epo
train
Bug Fix
512
N/A
AdamW
Linear
le-4
1.0
16
75
#Fine-tune
Vul Fix
512
N/A
AdamW
Linear
le-4
1.0
16
75
Prre-train
Bug Fix
512
256
AdamW
Linear
le-4
1.0
75
*Fine-tune
Vul Fix
512
256
AdamW W
Linear
le-4
1.0
8
75
A.2 BASELINE APPROACHES
We compare our proposed method VQR with the following basclines:
The vanilla transformer architecture is first pre-trained a bug-fixing corpus in a supervised manner, VRepair (Chen et al. (022) is then fine-tuned to fulfil the vulnerabil- ity repair task which shares a similar nature the bug-fixing task. We reproduce VRepair by following the instruction provided by Chen et al. to build and train the model.
VRepair:
The transformer encoder-decoder approach learns the representation of vulner- able functions and generates the corresponding repair patches. VulRepair (Fu et al. 2022) relies the CodeT5 (Wang et al. (2021b) model pre-trained through multi-task learning the Codesearchnet (Husain et al. 2019) dataset to learn the characteristics of programming languages. We reproduce VulRepair using the repository provided by Fu et al..
VulRepair:
TFix: Similar VulRepair (Fu al. 2022), TFix (Berabi et al. 2021) relies on trans-
former encoder-decoder model, however, pre-trained on natural language using denoising objectives (Raffel al. 2020) (i.e., T5-large). We reproduce TFix using the repository
provided by Berabi et al..
SequenceR: An RNN-based approach that learns the representation of vulnerable functions token by token, SequenceR (Chen et al.) 2019) leverages bi-directional LSTM encoders with unidirectional LSTM decoders. We reproduce SequenceR by following the instruction provided by Chen et al. to huild and train the model.
CodeBERT: A bi-directional self-attention pre-trained on the Codesearchnet (Husain et al., 2019) dataset using masked language modelling and replaced token detection ob- jectives (Feng et al.) 20200). Mashhadi & Hemmati] (2211) leveraged CodeBERT (Feng et al. 2020) for automated program repair of Java bugs and presented substantial improve- ment over RNN-based models. We reproduce CodeBERT using the repository provided by
Feng et al..
GraphCodeBERT: An extensive version of CodeBERT (Feng et al. 2020). GraphCode- BERT (Guo et al.) 202T) considers the Data Flow Graph (DFG) of code using the graph- guided masked attention during both pre-training and fine-tuning stages. We reproduce GraphCodeBERT using the repository provided by Guo et al..
Note we follow the best hyperparameter setting as reported by the original authors obtain the best results of each baseline method.
A.3 ANALYSIS OF OUR VQR's PERFORMANCE
We visualize the %PP across all CWE-IDs in our testing data as bar graph to explore our VQR ' performance for different CWE-IDs. addition, we show the frequency of each CWE- ID for both training and testing data as two line graphs explore the relationship between the frequency of samples and the performance of our method. Note that the ticks of the Y axis on the left for the &PP metric while those the right are for the data frequency of each CWE-ID..
12

---

Under review as a conference paper at ICLR 2023
100
Valin Fresenc Terpghuh
035
562 162-500
200 -000
2000
Su0 200
CREEIDS
Figure 6: The performance amalysis of our VQR based different vulnerability types (i.e., CWE- IDs). The bar chart represents the %PP while the blue line the training frequency and the red line the testing frequency across all vulnerability types. Note that the ticks of the Y axis the left for the %PP metric while those on the right are for the data frequency of each CWE-ID.
As shown in Figure 6 the performance of our approach varies for each CWE-ID. Our approach per- forms well on some of the CWE-IDs that all testing samples can be correctly repaired. Furthermore, we find that the frequency of training and testing samples are not highly correlated with the per- formance of our method. This indicates that automated vulnerability repair (AVR) is challenging problem in that high-frequeng samples may not guarantee the repair model's performance.
Table 5: The %PP of our VQR approach across the top 25 most dangerous CWE-IDs in 2022, The %PP shown based on the beam search results where Beam=5.
Rank
II
%.P
1
CWE-787
50% (13/26)
3
CWE-89
100% (2/2)
4
CWE-20
33% (24/72)
5
CWE-125
42% 13)
6
CWE-78
33% (1/3)
7
CWE-416
31% (9/29)
8
CWE-22
50% (1/2)
11
CWE-476
31% (11/36)
13
CWE-190
54% (19/35)
16
CWE-862
100% (1/1)
17
CWE-77
67% (2/3)
19
CWE-119
75% (223/296)
22
CWE-362
9% (3/34)
23
CWE-400
55% (11/20)
Average
55% (368/672)
To investigate whether our VQR approach can repair dangerous real-world vulnerabilities, we evalu- ate our approach based on the 2022 CWE Top-25 Most Dangerous Software Weaknesses released by the CWE community at pitttps:: / cwe. mi t re- org/ top25/ / i /
Eop25 htmi The results are presented in the above table. We find that our approach can correctly repair 55% of the vulnerable functions affected by the Top-25 most dangerous CWE-IDs, which is better than the average performance of our approach (i.e., 45.14%).
A.4 ANALYSIS OF OUR VQR'S ' VULNERABILITY QUERY AND MASK
To demonstrate that vulnerability query (VQ) and vulnerability mask can learn better cross- attention. We visualize the cross-attention scores between vulnerability queries and vulnerable source code representation. We compare the visualization for three approaches as follows:
13

---

Under review as a conference paper at ICLR 2023
Vul Query + Vul Mask (Our VQR method): Our proposed method that utilizes both vulnerability query (initialized based on repair tokens) and vulnerability mask.
Random Vul Query: The method that utilizes randomly initialized vulnerability query and vulnerability mask.
Without Vul Mask: The method that utilizes vulnerability query (initialized based on repair tokens) without using vulnerability mask.
In the example with vulnerable code area shown in Figure 7 the cross-attention score of our method has warmer colour in the vulnerable area highlighted by the red box. Thus, our method ac- tivates the rross-attention score between the vulnerable code representation and their corresponding vulnerability queries more than the other two variant approaches the vulnerable area.
In the example with vulnerable code areas shown in Figure g the cross-attention method has warmer colour in the vulnerable areas highlighted by the red box. Furthermore, our method has better contrast than the other two methods, in that the cross-attention scores between vuinerable code representation and their corresponding VQs are highly activated while the cross- attention scores between vulnerable code representation and non-corresponding VQs are slightly activated.
of our
These results indicate that our proposed VQ with the vulnerability mask method can guide vulnera- bility queries to attend to their corresponding vulnerable code areas.
Vul Query Vul Mask (Ours)
Random Vul Query
Without Vul Mask
Figure 7: The visualization of the cross-attention scores hetween vulnerability queries (axis X) and vulnerable source code representation (axis Y). The vulnerable scopes highlighted in red boxes. Our method on the left warmer than the other two, which activates the most cross-attention scores between vulnerable code representation and vulnerability query in the vulnerable scopes.
Vul Query + Vul Mask (Ours)
Random Vul Query
Without Vul Mask
readlly Sery
Figure 8: The visualization of the cross-attention scores between vulnerability queries (axis X) and vulnerable source code representation (axis Y). The vulnerable scopes are highlighted in red boxes. Our method on the left has the best contrast between cross-attention scores in vulnerable code areas and oon-uuleeeable code areas. In other words, the cross-attention activated by our method in the vulnerable scopes is higher than the other two methods while the cross-attention scores are lower than the other two methods when outside vulnerable scopes.
14


Reviews:
[{"review_id": "hKExHNDKHQV", "reviewer": "Reviewer_urCv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a transformer model for generating repair for software vulnerabilities. It trains a separate token-level classifier model to generate masks called vulnerability queries and incorporates these masks in both the encoder and decoder parts. Through experimental evaluation on Big-Vul and CVEFixes, it shows that this helps the model obtain better repairs compared to baseline models.", "summary_of_the_review": "The paper addresses a specific software engineering problem and makes a simple modification to transformers to encode domain knowledge. On one split of the test data, it gets a few percentage point improvement over baselines.", "strength_and_weaknesses": "Strengths\n-----\n* The problem of fixing software vulnerabilities is important in practice and paper performs evaluation on two of the most common vulnerability datasets against relevant baselines.\n* It achieves a few percentage improvement across different beam sizes.\n\nWeaknesses\n----\n* The modeling contribution of the paper is minimal. It incorporates a simple masking strategy that labels vulnerable code tokens and uses the masks in the self/cross attention computations.\n* The test set is split arbitrarily into 80-20 train+validation and test. The test set is small enough (5.4K samples) that a five fold experiment can be conducted. Can you report results on the entire set of examples similar to Table 2?\n* The training setup and accuracy of the vulnerability query mask prediction model are not reported.\n\nClarifications\n--------\n* If you are generating token-wise labels as query masks then why and how do you use global pooling (see the sentence before Eq 2)?\n* Which layers do you apply the query marks to in the encoder and the decoder?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PYW7l61SgU", "reviewer": "Reviewer_oYMK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper applies a Vision Tranformer (VIT)-based approach to automated\nvulnerability repair. The method is called Vulnerability Query based Software\nVulnerability Repair (VQR). The authors introduce vulnerability queries similar\nto the object queries of VIT and cross-match them with vulnerable code areas\nthrough a cross-attention mechanism to generate more accurate repairs. They also\nlearn a vulnerability query mask to focus on vulnerable code areas and include\nit into the cross-attention, and also into the self-attention in the encoder to\nlearn embeddings that emphasize the vulnerable areas of the program. Their\nmethod outperforms existing state-of-the-art methods based on a real-world dataset\nof 5417 vulnerabilities.\n", "summary_of_the_review": "I really liked the paper, but I have a serious concern about the vulnerability\nqueries which are at the heart of it. If I'm mistaken regarding this issue it\nprobably needs to be made clearer in the paper.\n", "strength_and_weaknesses": "### Strengths\n\nThe paper is based on a good idea, is very well written, and presents convincing\nresults with many comparisons to state-of-the-art and an ablation study.\n\n### Weaknesses and Questions\n\nMy main concern is that I'm not sure about the analogy between VIT and VQR,\nparticularly between object queries and vulnerability queries. Object queries\nare used by the model to predict objects in a given area of the image, are\nlearned throughout the training process (as an embedding), and are the same for\neach image (apart from being updated by learning). In contrast, it seems to me\nbased on 3.2.2 that vulnerability queries are just inputs to the decoder. This\nis reinforced by their initialization: they are initialized by the repair tokens,\nwhich are different for each example (they are the targets).\n\nIf I'm not mistaken then some of the parallels between VIT and VQR and some of the\narguments in the paper do not hold, and the state-of-the-art results are\nobtained probably because of the architecture from VIT and not the queries,\nwhich makes it necessary to rewrite significant parts of the paper, including\nthe title.\n\nI believe that a minus sign is missing from the cross-entropy loss (2). Also,\nmaybe it would be clearer to explicitly state that this is a cross-entropy loss.\n\n$x_i = [t_1, \\dots, t_n]$ is sometimes referred to as a sequence of tokens, and\nsometimes as a sequence of token embeddings.\n\nIt could be helpful to expand the VIT abbreviation in the paper.\n\nThere are some small typos:\n- in the Abstract, \"the real-world 5417 vulnerabilities\"\n- at the start of 3.1, \"Assuming\"\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GVwk8mhdm0", "reviewer": "Reviewer_Vf8F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at improving the transformer based vulnerable code repair techniques. Inspired by vision transformer based object detection approach, it proposes to incorporate a learnable  ‘vulnerability query mask’ into both encoder output and the cross-attention of existing transformer based models. Experimental results show the proposed approach provides an increased percentage of perfect predictions compared to state-of-the art approaches.", "summary_of_the_review": "The proposed approach does not demonstrate significant novelty. Also, the effectiveness of the proposed approach should be investigated more elaborately by examining the error types used for training and evaluation.\n", "strength_and_weaknesses": "Strength\n\n- Code repair is a challenging task and improving the accuracy of the transformer based models is a reasonable direction.\n- Ablation studies are performed to demonstrate the effectiveness of added vulnerability query mask.\n\nWeakness\n\n-  The improvement over the existing transformer based models proposed in this work has limited novelty. Proposed vulnerability is a direct mapping of the vision transformer setup.\n\n- The evaluation results do not discuss types of errors used for training and evaluation. It is also not clear if the base models using text-to-text transformers (e.g., TFix) are fine tuned with the same training dataset. What implementation of CodeBERT is used for the evaluation of vulnerability repair?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "57VX7Z6AWVe", "reviewer": "Reviewer_VFto", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve existing automated vulnerability repair problem by considering the prediction of precise parts of vulnerable code. By making an analogy to object detection problem in computer vision, they introduce the concept of vulnerability query counter to object query and adopt cross-attention mechanism to pay attention to vulnerable code areas. Besides, they also propose to learn an extra model to predict vulnerable areas, the prediction is used to construct a vulnerability query mask and further incorporated into both encoder and decoder’s self-attention. Evaluation is conducted on a read-world dataset which achieves outperforming results. ", "summary_of_the_review": "This work presents the key idea of providing code repair model with the precise location of vulnerable code and proposes two techniques: vulnerability query cross-match and vulnerability query mask. But the rationale and effectiveness of vulnerability query is not well explained and proved. And important baselines are also missing during the experimental evaluation. We believe that this paper can be further improved to be solid work.", "strength_and_weaknesses": "Strength: \n(1) The research problem is important among code repair community. \n(2) The idea of incorporating predicting vulnerable areas into code repair is interesting and the intuition makes sense.\n\n\nWeakness: \n(1) The concept of vulnerability query is somewhat weird as the authors cannot fully elaborate what it exactly is, especially when making analogy to object query. The vulnerability query is designed and utilized in the decoder part, how it functions is also not clearly described.\n(2) Some important baselines are missing and more metrics should be reported. There are a handful of works such as [1,2] which also use seq2seq models to predict program repair but these baselines are missing in experiments. Sequence based metrics should also be reported like BLEU, METEOR as they are widely used in previous code repair works.\n(3) The ablation study is not sufficient. While the authors highlight the importance of proposed vulnerability query, there is no ablation study to demonstrate the effectiveness of it.\n\n[1] Li, Yi, Shaohua Wang, and Tien N. Nguyen. \"Dlfix: Context-based code transformation learning for automated program repair.\" Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 2020.\n[2] Jiang, Nan, Thibaud Lutellier, and Lin Tan. \"Cure: Code-aware neural machine translation for automatic program repair.\" 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject