Suitable is the Best: Task-Oriented Knowledge Fusion in Vulnerability Detection
Authors: Jingjing Wang
Minhuan Huang
Yuanping Nie
Xiang Li
Qianjin Du
Wei Kong
Huan Deng
Xiaohui Kuang
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
Graph Neural Networks
vulnerability detection
static analysis
software security
knowledge fusion
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose KF-GVD, a knowledge fusion-based GNN model for task-oriented source code vulnerability detection.

Abstract: Deep learning technologies have demonstrated remarkable performance in vulnerability detection. Existing works primarily adopt a uniform and consistent feature learning pattern across the entire target set. While designed for general-purpose detection tasks, they lack sensitivity towards target code comprising multiple functional modules or diverse vulnerability subtypes. In this paper, we present a knowledge fusion-based vulnerability detection method (KF-GVD) that integrates specific vulnerability knowledge into the Graph Neural Network feature learning process. KF-GVD achieves accurate vulnerability detection across different functional modules of the Linux kernel and vulnerability subtypes without compromising general task performance. Extensive experiments demonstrate that KF-GVD outperforms SOTAs on function-level and statement-level vulnerability detection across various target tasks, with an average increase of 40.9% in precision and 26.1% in recall. Notably, KF-GVD discovered 9 undisclosed vulnerabilities when employing on C/C++ open-source projects without ground truth.

Suitable is the Best: Task-Oriented Knowledge Fusion in Vulnerability Detection
Jingjing Wang
Minhuan Huang"
Institute of Systems Engineering, Academy of Military Sciences, PLA jennywangel@163. com
Institute of Systems Engineering, Academy of Military Sciences, PLA darbean@126. com
Yuanpin Nie
Xiang Li
Institute of Systems Engineering, Academy of Military Sciences, PLA yuanpingnie@nin edu. cn
Institute Systems Engineering, Academy of Military Sciences, PLA ideal ideal_workdd wwrkw1r3........ com
Qianjin Du
Wei Kong
Department of Computer Science and Technology, Tsinghua University dqj dgj200mails. tsinghua. edu. cn
School of Information Science and Engineering, Zhejiang Sci-Tech University kong_ www.wewwwwwe.......... org
Huan Deng
Xiaohui Kuang
Institute of Systems Engineering, Academy of Military Sciences, PLA denghuan619@163 com
Institute of Systems Engineering, Academy Military Sciences, PLA xiaohui xiaohui_kuang@1.63 com
Abstract
Deep learning technologies have demonstrated remarkable performance in vulnera- bility detection. Existing works primarily adopt a uniform and consistent feature learning pattern across the entire target set. While designed for general-purpose detection tasks, they lack sensitivity towards target code comprising multiple functional modules diverse vulnerability subtypes. In this paper, present a knowledge fusion-based vulnerability detection method (KF-GVD) that inte- grates specific vulnerability knowledge into the Graph Neural Network feature learning process. KF-GVD achieves accurate vulnerability detection across dif- ferent functional modules of the Linux kernel and vulnerability subtypes without compromising general task performance. Extensive experiments demonstrate that KF-GVD outperforms SOTAs on dunetion-lever and statement-lewe vulnerability detection across various target tasks, with an average increase of 40.9% in precision and 26. 1 in recall. Notably, KF-GVD discovered 9 undisclosed vulnerabilities when employing on C/C++ open-source projects without ground truth.
Introduction
According to statistics, 26,447 vulnerabilities were disclosed in 2023, continuing the alarming trend of continuous growth in vulnerability numbers over the past seven years"] Static source code
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

-9256,
M8N 101% 3.55%
Net
restint, JuB jatesot soct TA4]
static 1at
distatic
GEC Ziee conoat pateo JomBE
M7%
11.8%
Fs
riteach remning Falen: donee
aff MeT_ desc 10887)
IELIST
Drivers
auces mhiokkin Ini
nicht 101A oet _tt___oe__2ll__ee_ JI7A P PIEEO
917%
Kernel
and 20 (
Block
Include
12 (Ccn Ten R3XI
Others
-251800 c26n2-1) 1837
Figure 1: The distribution of CWE-416 (left) and CWE-119 19 (right) vulnerabilities across all modules in the Linux kernel over the past decade.
Figure 2: CWB-416 vulnerability (left) and CWE-119 vulnerability (right) discovered in the net module.
vulnerability detection (VD), as an integral part of the software development lifecycle, plays a crucial role in enhancing system security and building reliable, high-quality software systems. Early static analysis tools required experts to define scanning rules for VD. Machine learning (ML)-based VD methods, on the other hand, required manual predefinition of code features. Currently, deep learning (DL)-based VD methods can achieve automated VD without the need for manual feature definition and have proven effective identifying potential vulnerability patterns. These studies can be categorized by the target of detection: (1) Targeting open-soure code projects. Most of the studies like IVDetect [ and Reveal 2 assess the performance of VD methods by evaluating their effectiveness across entire open-source projects, such as the Linux kernel, QEMU, and more. (2) Focusing on specific vulnerability types. Some studies primarily concentrate on specific commonly occurring vulnerability types in real-world applications, such as buffer overflow 3 use before initialization 4 use after free [5]. and so on.
With the prevalent of software applications, the scale and complexity of source code projects increase. On the one hand, the functionalities of various components or modules within project vary, leading to differences in the vulnerability triggering conditions and types. On the other hand, different triggering mechanisms result in a rich variety of subtypes within the same vulnerability type. Consequently, potential vulnerability patterns that may exist in target objects often closely correlate with specific source code contexts and characteristics of particular vulnerability types. However, the training objective of the current mainstream DL-based VD methods is to learn more comprehensive and effective vulnerability information, thereby obtaining an optimized model with the best generalization performance across the entire target source code dataset. In practical application, when generlized model is applied to specific scope of source code, the detection of more detailed and distinguishable potential vulnerability patterns becomes difficult, leading to a compromise in the effectiveness of the model. Additionally, the vulnerability uncertainty and complexity of the target detection object, as well as the limitations of available vulnerability data, pose challenges for transfer learning technology in achieving effective knowledge transfer between the source and target domains. As a result, the generalization capability of the transfer model in the target domain will be severely limited. In security practice, scalable models capable of achieving robust VD performance across both the source and target domains are anticipated. To this end, it is essential to design specific pattern learning methods for different target tasks.
In this paper, we propose KF-GVD, a Knowledge Fusion-based GNN model for source code Vulnerability Detection. The object of KF-GVD is to implement the most suitable vulnerabil- ity pattern learning and identification strategy for task-oriented VD in flexible manner, while maintaining a certain degree of generalization of the original model. By incorporating vulnerability knowledge into the feature learning process of the target, KF-GVD promotes the model to learn the vulnerability features that are closely related to the current task more efficiently. Through this way, VD methods in target-oriented vulnerability pattern learning
the biases of the current general-purpont can be corrected to some extent. KF-GVD employs a graph self-attention mechanism to capture high-weight nodes that influence model decisions during message propagation. By associating with code statements, it obtains fine- -rniinr vulnerability localization, achieving high transparency and interpretability of the model, thereby assisting developers in understanding the rationality of model decision-making. In summary, our contributions in this paper are:
We propose a task- knowledge fusion method, which integrates specific vulnerability knowledge in the feature learning process of GNN, it enables the seneral-purpose VD model to learn vulnerability features related the current target more effectively and achieve VD tailored to specific tasks.
2

---

Rxpreseatatiin
Phuse Valierabilitty Detrcttion Intergretation
Inpar
Ougaa
Sintowwn-Lend Inkegratativn
Croup EnonsAuue
CCN
Self-Altettion
Fork Ohieatted
Facernbatty Anowdedge
Graph Pooling
Grnph Gezeraton azd Knowledge Eunaction
Knowiedge Fesion
Classification
Powdiciion
Figure The overall architecture of KF-GVD.
We propose the framework KF-GVD, which can be flexibly adapt to target task detection while maintaining the generalization performance for source task. The interpretable KF- GVD achieves effectively vulnerability identification both the source code function level and statement level.
Extensive experiments demonstrate the superiority of KF-GVD in VD performance com- pared to SOTAs when targeting multiple functional modules diverse vulnerability subtypes. Note that KF-GVD discovered and submitted 9 undisclosed vulnerabilities in the open source C/C++ project without ground truth, proving its effectiveness in real-world applications.
2
Motivation
Figure D shows the distribution of the number of security commits related to CWE-416 (Use After Free) and CWE-119 (Buffer Overflow) in the Linux kernel over the past decade, collected from NVD. It can be observed that the occurrence proneness different types of vulnerabilities exhibits a significant module tendency, especially CWE-119. Of the 11 modules in the Linux kemnl,, more than 80% of the vulnerabilities occur in the first four modules with the highest number of vulnerabilities. Additionally, distinet triggering features for different vulnerability types the same module evident. As shown in Figure 2 presenting two typical patch examples in the network module. It can be observed from the code snippet on the left, leakage release is a major triggering form of CWE-416 vulnerabilities in the net module. In contrast, CWE-119 in the same module are usually related to restrictive checks lacking critical network protocol fields.
condition caused by pointer-related resource
The above examples are presented to demonstrate that in practical applications, the potential vulnera- bility patterns associated with program behavior differ and have distinct characteristics depending on the context of different detection targets and tasks. In such cases, the two existing types of DL-based VD methods, which employ uniform model optimization and analysis techniques for any target, are difficult make full of known information in diverse practical task scenarios characterize the potential vulnerability characteristics of different target codes. The greatly compromised detection performance will also further burden the code audit in the software development phase. To alleviate the above problems, KF-GVD employs simple and efficient knowledge guidance form of artificial "prompt", which enhances the rationality of model pecisionnmakinne by combining the historical information of target tasks and human prior knowledge, corrects the deviation of pattern learning to certain extent, and makes more flexible to adapt downstream tasks. As the first task-oriented GNN- VD method, KF-GVD strives to achieve more accurate and scalable VD with lower costs.
3
The KF-GVD Framework
The overall architecture of KF-GVD is illustrated in Figure 3 KF-GVD consists of two main phases: feature representation and vulnerability detection and interpretation.
3.1 Feature Representation
In the feature representation phase, the source code initially transformed into Code Property Graphs (CPGs) the intermediate representation. Subsequently, task-specific vulnerability knowledge

---

stzuct
-20 coud co
Port Siurce
Du
F
Fki
[labll
(assiog stslopy
Fl,
[aabll
dt
dg
Kaomlodge Fueion
f
WordWVor
K
Yauw
Vereaeetle
Por Torget
Figure 4: An example of feature representation.
Figure 5: The workflow of KF-GVD.
subgraphs extracted from the CPGs. Finally, node feature vectors encoded according to the node information of the graph, and combined with the adjacency matrix to form graph embedding as the input of the model.
3.1.1 Code Property Graph Generation
The concept of CPG proposed by Yamaguchi, et al. (2014) [6] is joint data structure that combines abstract syntax tree, control flow graph, and program dependency graph. CPG has been shown to model more diverse types of vulnerabilities than conducting combination of single or two code properties as an intermediate representation insight behind KF-GVD's s adoption of CPG as the intermediate representation is to comprehensively preserve source code information in this stage. This not only enhances the performance of the model in detecting vulnerabilities across the entire project, but also lays the foundation for the model to characterize more detailed vulnerability features in subsequent target tasks.
the source code for static analysis 7 8]. The key
3.1.2 Task-oriented Vulnerability Knowledge Extraction
KF-GVD extracts the knowledge subgraph by initially screening the codes with the most feature relevance the current task target, thus affecting the optimization from the source task generalization model to the specialized target expert model. The task-oriented vulnerability knowledge considered by KF-GVD includes: vulnerable program operations and sensitive functions related the vulnerability type, as well as customized knowledge for specific tasks associated with the target functional scenarios. As for implementation, we first perform coarse screening of source code statements for specific task objectives based on relevant vulnerability knowledge. Then, we map the identified statements node sets in the CPG and mask out the remaining nodes weakly related the task. The left part of the feature representation example in figure 4 shows the extraction process of the task-oriented vulnerability knowledge subgraph. For CWE-119 VD in the file module of the Linux kernel, because the code statement (highlighted in orange) in the code snippet calls the strlepy() function, which considered be sensitive function that may introduce CWE-11 SWE-119 type vulnerabilities under the current task, the associated node (highlighted in orange) and edges of this statement in CPG is extracted as the knowledge subgraph.
Appendix A provides a more detailed description of the task-oriented vulnerability knowledge considered by KF-GVD.
3.1.3 Graph Embedding
The CPG node information generated by Joem? comprises two parts: code element types and code statements. Figure shows the corresponding information for nodes v&i and u12 in the CPG. KF- GVD adopts two vectors, Vap and funze describe the code element types. Vop reflects the operation type of the code associated with the node, such as field access, memory allocation, mathematical operations, and on. Vjune indicates special function calls, code field types, and on. Both Vap and Vrunc use one-hot encoding, and when a node contains only one type of information. the remaining vector is zero-encoded. Next, we employ pretrained Word2 model to map tokens
https://joermii

---

generated from code statements to fixed-length feature vectors to generate V Veemantic that represent the semantic information of the source code corresponding the node. Finally, the three vectors are concatenated to form the final feature vector of the node. The spatial features of CPG represented by the adjacency matrix and the node feature matrix constitute the graph embedding, serving as the input for the subsequent model stage.
3.2 Vulnerability Detection and Interpretation
In this section, we provide detailed description of the GNN model adopted in KF-GVD and explain how vulnerability knowledge is leveraged to optimize the model for specific subtasks during VD. Finally, we describe how KF-GVD utilizes attention mechanisms interpret and locate potential vulnerability source code.
3.2.1 Object
For the source code function C corresponding the CPG = (V,E), G. we define y E ey as the function-level label of G corresponding to C, where Y = 10.1). = 1 indicates that the sample is a vulnerability sample, y = otherwise. We define source task O corresponding to dataset Do = [(G J1 E 9o. y E Yo N Target task T corresponds to dataset DT = {(G,, 3r) G, E GT, y4 E Yr) where N and M are the total number of samples contained in Do and Dr. and N > M. Assuming that a mapping FD. Go + Vo has been learned on the source task dataset Do. KF-GVD aims to establish a mapping f : G. - Y on the target task T with the assistance of knowledge K2 relevant to task t. Here, d, represents the dataset associated with any subtask t under the target task T. Therefore, the workflow of KF-GVD is mainly divided into two parts: model training on the source task and knowledge fusion the target task. The workflow of KF-GVD is shown in Figure 5
3.2.2 Task-oriented Model with Knowledge Fusion
The second phase in Figure 3 shows the general hierarchical structure of the GNN employed by KF-GVD. The model utilizes two layers of Graph Convolution Network (GCN) to calculate the new representation of each node by weighted summation of neighbor node information. Subsequently, based on the self-attention mechanism, the model obtains weights between nodes, calculating and retaining high-weighted nodes during message propagation (9]. The graph pooling layer is then employed to obtain the embedding of the entire graph. Finally, the model obtains function-level vulnerability identification results for the source code based on the classification layer. Furthermore, the training process of the model in the source task stage is similar to the current DL-based VD method. On this foundation, when conducting VD tasks for specific targets, a task -rienteee knowledge fusion layer is introduced.
The current state of the graph G = (V, F), obtained from the GCNs layer, is denoted Hy (he by E V},j E n.). and the corresponding knowledge subgraph G2 = (V&, Ex) on the target task t is denoted as Hv. = {hn, U. E Vah.g E [I...,m. The calculation of the state Hu = [ha,lo, l0, E V}.j {l,..., n) for G (V, E) in the fusion layer can be defined as follows:
hug {{
Fusion(aku,, Bhu,)) us E Vv hey, Uy & Vv:
where a and 33 are fusion coefficients, reflecting the degree to which knowledge tailored for a specific task / influences the pattern learning of the model. Assuming that in the source task O the model has been trained on dataset Do, denoted as FD. The training model f for a subtask i, t E T includes the following steps:
(1) Dataset collection: Define task t corresponding to dataset de Dr. To ensure that the fused model if retains the ability to detect vulnerabilities in O to some extent, the training set due for is derived from de and random sampling of Do, defined as do E Doo d = d, + d.
(2) Initialization: Initialize the parameters of f using Fro to achieve knowledge sharing between the source task and the target task.
(3) Fusion: During the training phase, we perform the feature fusion operation (Equation I only on the data d' randomly sampled from dy. The fusion function employs a weighted sum of node features.
5

---

For implementation, we first evaluate the detection performance of FDo on di and consider the misclassified samples in the first round by FD. as the preferred subset of dy for the fusion operation in the training process of f. We treat this subset of samples as the data subset with understanding bias of FDg the target task. Overall, the knowledge fusion process employed by KF-GVD also process of manual correction and adjustment carried out during the model training phase.
more detailed computational description of each layer of the model is provided in Appendix B.
3.2.3 Statement-level Interpretation and Location
KF-GVD achieves interpretability through the self-attention mechanism. In the self-attention layer, the top [m/] nodes with the highest attention scores in G will be retained the input for the pooling layer, and the states of the remaining nodes will be masked during subsequent message propagation, where 7 the retain rate. The source code statements corresponding the crucial nodes which have high influence on f to make vulnerability decisions are regarded as the interpretation of the current sample, that is, the fine-grained vulnerability statement location. In the implementation, we realize the mapping from graph nodes to source code statements through the data files generated by joern.
To improve the performance of KF-GVD in function-level and statement-level VD, we define graph G = (V, E) with a labeled node set A = [(,,,4)) f/y E V, E ")>==> where I represents the node label. The prediction function f for the target task & can be learned by minimizing the following loss function:
min Σ
((Fo(((()))
j=l
Here, [dpl represents the size of the training dataset under task I, is an adjustable hyperparameter.
Evaluation
We evaluate the superiority of KF-GVD compared to baseline approaches two source tasks, S119 and S416, as well as their corresponding target tasks, Tm and Teub. in order to answer the following research questions:
RQI: How does KF-GVD perform compared other function-level VD methods?
RQ2: How does the interpretation of KF-GVD compare to other fine mre-ggained VD methods in terms of locating vulnerable statements?
RQ3: How does KF-GVD perform in detecting and locating vulnerabilities real-world software products where the presence of vulnerabilities is unknown?
4.1
Experiment Settings
4.1.1 Dataset
The evaluation dataset consists of two parts: the source task dataset and the target task dataset. Following the current practice of DL.-based VD methods, we initially train the model on a widely collected source task dataset to obtain a model with good generalization performance for a specific vulnerability type. Subsequently, the model is applied to the target task dataset for VD.
Source Task Dataset: We define two source tasks targeting the detection of CWE-119 and CWE-416 type vulnerabilities, denoted as S119 and S116 respectively. The source task dataset consists of 80% CWE-119 and CWE-416 type vulnerability information extensively collected from 13 real-world C++ projects from N The remaining 20% is sourced from academic security defects and synthetic data provided by SARDT
Referring to the types of research objects in the current DL-bbaed VD studies (Section 1), our target tasks are divided into two categories: (1) Tm Detection of CWE-119 19 and CWE-416 type vulnerabilities within various modules in the Linux kernel, denoted as T Tm.. and Tmasg respectively. (2) Taus: Detection of CWE-119 subtypes, specifically CWE-125 and CWE-787.
Target Task Dataset:
3 "httss//nvd.nist..oom https://aamtteiiissgg///.ttttsss

---

Table I: Comparison of function-level VD of T mm and Tsub S119. P: Precision 96); R: Recall(%);
F1: F1-score( %)
S119
Method
Fs
Drivers
Net
Include
CWE-123
CWE-187
F1
P
F1
P
F1
P
F1
F1
R
F1
R F1
Oppcheck
45.0 55.7
49.8
33.7
50.5
40.4
32.1 45.9
37.8
44.2
40.0
25.9 28.6
24.8 50.6
33.3
29.4 35.7 32.2
Flawfinder
27.6 50.4
35.7
15.3
57.4
24.2
25.9
44.8
32.8
37.6
42.642.840 40.0
29.7 56.8 39.0
12,9
37.4
19.2
18.3 33.5 23.7
Sysver
54.8 70.6
61.7
23.6
67.2
34,9
28.1
37.6
15.7
25.0
33.0
37.2
39.7
47.3
33,4
19.6
VulCNN
63.9 77.4
70.0
35,5
50.7
41,8
27.8 44.6
34,3
39.4 58.6 47.1
22.0 29.2
16.8
29.1 21.3
17.6 23,0
Codebert
63.2 67.9
66.3
54.7
39.5 45.9
37.5 40.0 38.7
48.5
44.1 46.2
34.6
51.8414411 41.5
51.8
34,8
57.6 43.4
43.7 48.6 46.0
CodeLlama
70.0
55.7
45.6
45.7
57.2
52.2
49.3
$1.5
37.6
44.3
51.6
Wizardeoder
72.4 52.4 60.8
62.5 15.5 45.3
45.8 48.7 47.2
50.8
42.0 46.0
48.6 56.6 52.3
33,5 52.5 40,9
47.5 51.9 49.6
Devign
68.5 70.2 69.3
30.6
54.2 39.1
35.4 42.8 38.7
48.6 52.6
25.8 40.3 31.5
20.1 37.9 26.3
18.4 25.0 21.2
ReGVD
74.1 71.2 72.6
60.8
34.2 43.8
40.9 47.1 43.8
52.1
59.1 55,4
44.1 50.8 47.2
29.8 54.0 38.4
44.9 57.2 50.3
IVDetect
79.0 83.3
81.1
46.7
13.3 38.9
33.3 66.7 44.4
66.7
50.0 57.1
40.0 46,2 42.9
31.9 55.8 38.
46.8 52.4 43.0
GVD.B
82,9 90,9
73,5,58.7
65.2
66.7
26.2
76.3 64.7
$7.1
59.3
49.8 60.5 54.6
66.7 61.5 64.0
KF-GVD
82.9 90,9 86.7
96.1 95.2 95.7
90.0 94,77 92.3
91.7
75.0 82.5
91.7 84.6 88.0
59.2 80.0 67.9
80,0 84.2 82.1
Table 2: Comparison of function-level VD of Tman on S416- P: Frecision(%) R: Recall Fl:
Fl (l-score(%)
Tm
Method
S418
Net
Drivers
Kernel
Block
Include
F1
P
R
FL
P
F1
F1
R
F1
R F1
R F1
Cppcheck
27.7 42.6
33.6
22.7
17.9
27.0 35.9
30.7 45.9 36.8
10.3 45.9
16.8
30.2 36.5
33.1
23.6 31.8 27.1
Flawfinder
33.4 45.9
38.7
20.6
15.9 23.2
9.0
28.5
39.2
17.8
21.4
25,0
30.7
Sysver
58.4 67.2
62.5
21.9 40.5 28.4
27.3 31,5
52.5
32.5 31.6
22.7 23.6 23.1
37,9 30.2 33.6
26.3 45.7 33.4
VulCNN
66.9 72.8
69.7
33.4 52.7 40.9
42.0 49.6
28.5
48.1
34.3
36.8 56.3 44.5
24.7 65.1
35.8
22.5 39.6 28.7
Codebert
50.3
47.8
41.5
51.6 46.5
46.5 51.1 48.7
42.9
41.8
40.9
CodeL.lama
65.9 59.1 62.3
52.9 46.0 49.2
50.6 52.6 51.6
44.1
43.9 44.0
53.3 $1.5 52.4
40,5 41.1 40.8
$7.6 52.8 55.1
Wizardeoder
59.6 69.3
64.1
53.7 48.2 50.8
42.7 38.2 40.3
39.8
52.0 45.1
55 13.650.5 52.9
44.3 38.5 41.2
56.4 49.3 52.6
Devign
63.7 79.4 70.7
37.1 42.6 39.7
48.9 50.2 49.5
34.1 56.9 42.6
37.5 44.6 40.7
48.1 33,9 39.8
36.8 70.4 48.3
ReGVD
67.2 71.7 69.4
41.9
43.7 42.8
50.3 $1.5 50.9
40.6
45.9 43.1
45.8 55.5 50.2
42.8
34,8 38.4
44.5 65.2 52.9
IVDetect
81.8
87.8
40.4
38.2
$1.7
51.8
43.8
42.4
41.4
49.0
39.0
37.2
66.7
72.7
GVD-B R
86.8 89.3
88.0
53.6
88.2 66.7
73.3
66.0 69.5
41.3
44.7 42.9
39,5 44.7 42.0
$1.7
78.4 51.9
50.0 53.3 $1.6
KF-GVD
86.8 89.3 88.0
78.6
98.1 87.3
73.9 94.4 82.9
87.1 71.8 78.7
85.4 92.1 88.6
66.7
83.3 74.1
82.4 93.3 87.5
Both of these vulnerability subtypes are also included in the 2023 CWE Top 25 Most Dangerous Software Weaknesse
4.1.2 Baseline Approaches and Evaluation Metries
We compare KF-GVD at the function level with four types of VD methods, including: (1)sule-based
commercial static code analysis tools: Cppcheck and Flawfinder (2) Classical DL-based VD
methods: Sysver [10] and VulCNN [T. (3) Large-scale code models: Codebert 12]. Code Llama
[3], and Wizardcoder [14. (4) GNN-based VD methods: Devign [8), ReGVD [5] and IVDetect [[]. For statement-level localization comparison, we chose the state-of-the-art fine-grained VD methods
IVDetect M. Vul 16 and LD 17 as baselines.
We evaluate the performance of KF-GVD and the baseline methods using Precision (P), Recall (R), and Fl-score (F) for both function-level and statement-level comparison. Besides, also introduce the ranking metric Mean Average Precision (MAP) in order to explore the performance of the method on code statements that it considers as the best interpretation for function prediction. Appendix C provides more specific information on the datasets and experimental settings adopted by KF-GVD.
4.2 Function-level Vulnerability Detection Performance
For ML-based VD methods in addition to large-scale code models, we train and test all models on the source tasks S119 and S116 separately, and then apply them to the corresponding target tasks. Additionally, following the concept of transfer learning, we employed the same GNN model as KF-GVD in the source tasks. For VD in the target tasks, we fine-tuned the GNN model used in KF-GVD, denoted as GVD-ft, instead of employing knowledge fusion operations, so the results of VD on the source task are the same as KF-GVD.
'httss//wwwe.nn www..//......tttt/eetiiiiiiiiiii 14tss//14452.. html "nttp://oppccheck.eet http::/dwhheeler.cm//ttttieeee

---

Table 3: Comparison of statement-level VD of m11 and Tub S119. P: Precision( (%); R:Recall
Fl: F1-score(%6
112
Toub
Method
F5
Drivers
Net
Include
CWE-125
CWE-187
R
F
P
R
F
P
IVDetect
32.3
56.1
34.8
10.5
63.1
15.4
36.7
20.4
26.0
9.7
74.7
16.4
2.2
17.1
3.1
16.7
10.0
12.5
LineVD
39.
27.9
32.6
11.0
58.7
16.1
37.6
21.2
26.8
17.2
26.1
4.1
24.9
53
33.3
20.0
25.0
LineVul
33.8
45.0
38.6
10.7
24.0
14.8
22.4
28.0
24.9
16.3
44.8
23.9
6.4
13.6
8.7
29.8
19.0
23.2
GVD-Rt
32.1
55.0
34.5
112
66.0
16.4
18.3
10.2
13.0
9.6
85.4
16.3
7.5
51.0
10.3
2.9
18
2.2
65.5
Table 4: Comparison of statement-level VD of Timi on S116. P: Precision(%)) R: Recall(%); Fl: F1 (l-score(%)
Tmeis
Method
Net
Drivers
Kernel
Block
Include
R
F
P
R
F
P
R
F
P
R
F
P
R
F
P
R
F
Deteet
19.6
58.1
24.5
13.4
19.2
20.2
77.9
21.7
83.8
36.9
15.4
G7.9
67.7
67.3
LDneVD
24.0
98.8
31.3
16.6
55.9
25.6
17.9
75.2
23.3
15.8
72.9
21.9
12.5
16.7
14.3
48.2
49.2
48.7
Vul
20.9
45.3
28.6
15.3
44.0
22.7
22.8
32.8
26.9
24.9
41.8
31.2
14.1
48.0
21.8
31.7
36.4
33.9
GVD-ft
25.3
16.7
22.4
10.8
KF-GVD
56.3 96.3
63.8
55.9
80.8
66.0
76.5
81.1
68.1
80.6
75.9
75.1
27.4
97.3
36.1
733
73.1
72.6
Table and Table [2 show the comparison of function-level VD performance on different source tasks and their corresponding target tasks. It can be obviously observed that despite all baseline methods being designed to detect the same type of vulnerabilities, they exhibit noticeable decrease in performance metries across various specific target tasks. In contrast, KF-GVD consistently performs well. Compared to the best results among baseline methods (indicated by underlines in the table), KF-GVD demonstrates an improvement in precision by 0.6%-44%. recall by 5.8%-29. 29.62999%% and an average gain of 22.6% on F1- Fl-score. Although GVD-ft achieved relatively better results than other baseline methods most target tasks due fine-tuning of the model, GVD-ft shows 23.3% lower F1 average Tm a 31. 31.6% lower Fi-score average on T m4116 and 15.7% lower F1-score average on Toub compared to KF-GVD. This further proves the flexibility and effectiveness our approach in VD tailored to specific tasks.
Overall, the effectiveness of rule-based static analysis tools is inferior to that of ML-based VD methods. This is due to the limitations of finite and fixed manually predefined static scanning rules when dealing with multiple types of vulnerabilities and detection targets. Moreover, methods that adopt models like LSTM and CNN exhibit an average reduction of 20.6% on Fi-score compared to approaches based on GNNs. some extent, this reflects the powerful spatial feature learning capability of GNNs when dealing with structured languages like source code, opposed to the flattened feature processing approach classical DI. networks. It worth noting that the emerging large-scale code models have mediocre performance in dealing with tasks related to code vulnerability detection, and the F1 -c-oros on the corresponding target tasks of $119 and S416 are on average 34.3% and 33 .44 lower than those of KF-GVD. We infer that this is because current large language models mainly focus on question and answer, completion and other generative tasks rather than classification tasks.
Statement-level Vulnerability Detection Performance
Table 3 and Table present the comparison of statement-level vulnerability localization results between KG-GVD and other baseline methods on the target tasks corresponding to S119 and S three different target tasks, KF-GVD ex- hibits a precision improvement of 5.4% 52.9% and an average recall enhancement of 2.6%. vulnerability localiza-
= IVDetect
80
LineVD
(AP@5(%)
LineVul
GVD-ft
30
KF-GVD
73.7% in statement- tion compared to suboptimal detection results. This corresponds to an average gain of 59.7% in precision and 30.9% recall, showcasing KF- GVD's robust coverage and localization capa- bilities for vulnerable code during fine-grained detection. It is worth noting that GVD-ft, which
T Tm1119
T
Tm4116
Figure 6: Statement-level VD comparison on MAP@5.

---

performs suboptimally at the function level, does not exhibit equally good results at the statement level. This indicates that the fine-tuning approach employed by GVD-ft does not accurately capture the underlying vulnerability patterns of the target task. Moreover, MAP reflects the average precision
on the top K confident predictions. Following established research practices [ [8], we set K=5
and compare the statement-level MAP across all methods, as illustrated in Figure 6 Observably, KF-GVD consistently outperforms other methods in MAP on all target tasks, particularly achieving an average improvement of 42.4% on Tm119 which further proving the rationality of the function-level predictions from KF-GVD.
4.4 Case Study
We applied KF-GVD to various C++ open-source libraries commonly targeted for vulnerability detection. For projects without ground truth, we initially gathered information on publicly disclosed vulncrabilities for the target objects (obtained from GitHub) and performed detection for specific unknown vulnerabilities, particularly focusing on several high-risk issues. Taking multiple versions of Assimp (Open Asset Import Library), used for importing various 3D model file formats, as an example, we conducted detection on a total of 312 C++ source code files under the Assimp's s assetlib directory. The detection revealed 19 vulnerabilities, including three reported and published CVE.2222-445748, CVE-2021-459445 security issues, and 2 undisclosed vulnerabilities that have submitted to CNNVD) and been confirmed. Table presents all the undisclosed vulnerabilities detected by KF-GVD in different C++ open-source objects.
vulnerabilities (CVE-20222388522
pending confirmation
Table Undisclosed vulnerabilities detected by KF-GVD different C++ open-source objects.
II)
Project
File Location
Vul line
CNNVID-003--4377715
/OpenDDDL
348
assimp
CNNVID-2033-1259944
/FBParrer.cppp
192
CNNVD-2023-5996687
/ /detailrappddmml.pp
644
boost
CNNVD-2023--23891133
7 /basic (basic_reeex_____ _etrcttccrrrrrpppppppp
710
CNNVDD-00332200013
c-blosc2
/blosc-private.h
120
CNNVD=20233-76730992
exiv2
/value.cpp
13
BBNVD-2023-90736133
flatbulfers
/util.h
133
CNNVD-2023-838811.
frr
/bgp _ttr.c
2658
CNNVD-2023-27700
harfbuzz
1. /hb-atomic.hh
172
Disscusion
5.1
Threats Validity
First, in practical applications, we find that KF-GVD has more obvious advantages in target tasks with more historical version iteration information and knowledge related specific vulnerability types, which indicates that the performance of the method can be continuously improved with the increase of target task-related vulnerability knowledge. However, without any historical vulnerability information about the target software or knowledge about a specific vulnerability type, method performance degrades to the level of vulnerability detection with the generalized model. Second, we only verify the validity of the method on C/C++ code, not other programming languages, and in principle the method can be extended. Third, we only focus on vulnerability identification and fine-grained localization within functions, and cannot detect source code vulnerabilities that have cross-function or cross-file dependencies.
5.2 Limitations and Future Work
During the data processing stage, generating CPG data from source files accounts for more than 75% of the total processing time, highlighting the need more efficient data generation tools and process- ing strategies in the future. Additionally, in the feature embedding stage, we truncate the Vende that exceeds the length threshold of the feature vector, which leads to the loss of semantie information to certain extent. In the next phase, leveraging the powerful representation and generation capabilities

---

of current large-scale code models to obtain more comprehensive and efficient source code features will be a future research direction to enhance vulnerability localization performance. Finally, while KF-GVD has significantly improved statement-level vulnerability localization compared to SOTAs, there remains substantial room for improvement in fine-grained localization precision relative to function-level detection results.
More ablation studies on vulnerability knowledge sensitivity, feature representation, and experiments on cross-domain tasks are provided in Appendix D.
6 Related Works
Current DL-based VD methods achieve automated VD, alleviating the manual burden associated
with rule-based [99 and ML-based approaches 20 21 22 23 24]. VulDeePecker [3], Sysevr [10),
WVuleeePecker (25. VulCNN [1]] combined with classical DL models such as LSTM, BGRU,
CNN, are employed perform VD on various open-source projects or specific vulnerability types.
The detection granularity of these methods is at the slice or function level. Additionaly, Devign [8]. BGNN4VD [26). Reveal (2) and many other studies based on GNNs perform function-level VD on entire projects such as QEMU, FFmpeg, Linux kernel, or mixed datasets.
In recent years,
studies have achieved fine-grained vulnerability localization. VulDeelocator 27 utilizes intermediate code to define program segments for VD, accommodating semantic information that cannot be conveyed by source code-based representations. By employing the idea of granularity refinement, VulDeelocator outputs a finer granularity than its input, enhancing the precision of the detector. Furthermore, IVDetect [1] considers vulnerable statements and their surrounding context separately through data dependency and control dependency, enabling the model to better distinguish vulnerable statements. Additionally, IVDetect introduces GNNExplainer (28) provide subgraphs in program dependency graph (PDG) as explanations, containing key statements related to the detected vulnerabilities. Besides, VD (18] detines statement-level vulnerability detection as a node classification task, utilizing GNNs to leverage control and data dependency relationships between statements. By resolving conflicts between function-level and statement-level information and learning from both levels, LD significantly improves performance.
7 Conclusion
In this paper, we propose KF-GVD, a knowledge fusion-based vulnerability detection method. KF- GVD alleviates the limitations of current general-purpose detection methods when applied to the contexts involving multiple functional modules or diverse types of vulnerabilities. By integrating task- oriented vulnerability knowledge. KF-GVD prompts the model efficiently explore vulnerabilitiy patterns tailored to specific tasks while still maintaining general task performance. Our empirical evaluations demonstrate the superior performance of KF-GVD tailored for diverse specific tasks in both function- and statement-level VD. The case study we conducted on real C++ open- source projects further substantiates the practical effectiveness of KF-GVD in real-world applications. Notably, KF-GVD identified 9 undisclosed vulnerabilities when applied to real-world C++ open- source projects, further proving its practicality.
References
[1] Yi Li, Shaohua Wang, and Tien N. Nguyen. Vulnerability detection with lnn-grained inter- pretations. In Proceedings of the 29th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2021, page 292-303, New York, NY, USA, 2021. Association for Computing Machinery.
[2] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet. IEEE Transactions on Software Engineering, pages 1-1, 2021.
[3] Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. Vuldeepecker: A deep learning- system for vulnerability detection. arXiv preprint arXiv:1801.01681, 2018.
10

---

[4] Yizhuo Zhai, Yu Hao. Hang Zhang, Daimeng Wang, Chengyu Song. Zhiyun Qian, Mohsen Lesani, Srikanth V. Krishnamurthy, and Paul Yu. Ubitect: a precise and scalable method to detect
bugs in linux kernel. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2020, page 221-2321 New York, NY, USA, 2020. Association for Computing Machinery.
[5] Jia-Ju Bai, Julia Lawall, Qiu-Liang Chen, and Shi-Min Hu. Effective static analysis of con- currency Use-After-Free bugs linux device drivers. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 255-268, Renton, WA, July 2019. USENIX Association.
[6] Fabian Yamaguchi, Nico Golde, Daniel Arp. and Konrad Rieck. Modeling and discovering vulnerabilitiie with code property graphs. In 2014 IEEE Symposium on Security and Privacy, pages 590-604, 2014.
[7] Fabian Yamaguchi. Pattern- based vulnerability discovery. 2015.
[8] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances neural information processing systems, 32. 2019.
[9] Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International conference on machine learning, pages 3734-3743. PMLR, 2019.
[10] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing, 2021.
[11) Yueming Wu, Deqing Zou, Shihan Dou, Wei Yang, Duo Xu, and Hai Jin. Vulenn: an image- inspired scalable vulnerability detection system. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, page Association for Computing Machinery.
New York, NY, USA, 2022.
[12] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprini arXiv:2002.088155 2020.
[13] Wenhan Xiong Grattafiori, Alexandre Défossez, Jade Copet. Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[14] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08555 2023.
[15] Van-Anh Nguyen, Dai Quoe Nguyen, Van Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. Regvd: revisiting graph neural networks for vulnerability detection. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE '2, page 178-182, New York, NY, USA, 2022. Association for Computing Machinery.
[16] Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulner- ability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 608-620, 2022.
[17] David Hin, Andrey Kan, Huaming Chen, and M. Ali Babar. Linevd: statement-level vulnerability detection using graph neural networks. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, page 596-607, New York, NY, USA, 2022. Association for Computing Machinery.
[18] David Hin, Andrey Kan, Huaming Chen, and M Ali Babar. Linevd: Statement-level vulnerability detection using graph neural networks. arXiv preprint arXiv:2203.0518/ 2022.
11

---

[19] Morteza -aasrri-aasaaadii Saeed Parsa, Mohammad Ramezani, Chanchal Roy, and Masoud Ekhtiarzadeh. A systematic literature review on source code similarity measurement and clone detection: Téchniques, applications, and challenges. Journal of Systems and Software, 204:1 11 1179, 2023.
[20] Boris Chernis and Rakesh Verma. Machine learning methods for software vulnerability detection. In Proceedings of the Fourth ACM International Workshop on Security and Privacy Analytics, IWSPA '18 page 31-39, New York, NY, USA, 2018. Association for Computing Machinery.
[21] Guoyan Huang, Yazhou Li, Qian Wang. Jiadong Ren, Yongqiang Cheng, and Xiaolin Zhao. Automatic classification method for software vulnerability based on deep neural network. IEEE Access, 7:28291-28221 2019.
[22] Riccardo Scandariato, James Walden, Aram Hovsepyan, and Wouter Joosen. Predicting vul- nerable software components via text mining. IEEE Transactions on Software Engineering, 40(10):993-1000 1006, 2014.
[23] Bo Shuai, Haifeng Li, Mengjun Li, Quan Zhang, and Chaojing Tang. Automatic classification for vulnerability based on machine learning. In 2013 IEEE International Conference on Information and Automation (ICIA), pages 312-318, 2013.
[24] Istehad Chowdhury and Mohammad Zulkernine. Using complexity, coupling, and cohesion metries as early indicators of vulnerabilities. Journal of Systems Architecture, 57(3):294-313, 2011. Special Issue on Security and Dependability Assurance of Software Architectures.
[25] Deqing Zou, Sujuan Wang, Shouhuai Xu, Zhen Li, and Hai Jin. pvuldeepecker: A deep learning-based system for multiclass vulnerability detection. IEEE Transactions on Dependable and Secure Computing, 1865):2224-2236, 2021.
[26] Sicong Cao, Xiaobing Sun, Lili Bo, Ying Wei, and Bin Li. Bgnn4vd: constructing bidirec- tional graph neural-network for vulnerability detection. Information and Software Technology. 136: 106576, 2021.
[27] Zhen Li, Deqing Zou, Shouhuai Xu, Zhaoxuan Chen, Yawei Zhu, and Hai Jin. Vuldeelocator: a deep bearing-basse fine-grained vulnerability detector. IEEE Transactions on Dependable and Secure Computing, 2021.
[28] Zhitao Ying. Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnex- plainer: Generating explanations for graph neural networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[29] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1609:::600.0000000 2016.
[30] Xu Duan, Jingzheng Wu, Shouling Ji, Zhiqing Rui, Tianyue Luo. Mutian Yang, and Yanjun Wu. Vulsniper: Focus your attention to shoot fine-grained vulnerabilities. In IJCAI, pages 4665-4671, 2019.
[31] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
[32] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 17 10903, 2017.
[33] Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Yang Xiang, Olivier De Vel, and Paul Mon- tague. Cross-project transfer representation learning for vulnerable function discovery. IEEE Trunsactions on Industrial Informatics, 14(7):3289-322 2018.
[34] Zimin Chen, Steve Kommrusch, and Martin Monperrus. Neural transfer learning for repairing security vulnerabilities code. IEFE Transactions on Softwrre Engineering,499 49(1):147-165, 2023.
12

---

[35] Ashita Diwan, Miles Q. Li, and Benjamin C. M. Fung. Vdgraph2vec: Vulnerability detection in assembly code using message passing neural networks. In M. Arif Wani, Mehmed M. Kantardzic, Vasile Palade, Daniel Neagu, Longzhi Yang, and Kit Yan Chan, editors, 2 IEEE International Conference on Machine Learning and Applications, ICMLA 2022, Nassau, Bahamas, December 12-14, 2022, pages 1039-1046. IEEE, 2022.
[36] Jiao Yin, MingJian Tang. Jinli Cao, and Hua Wang. Apply transfer learning to cybersecu- rity: Predicting exploitability of vulnerabilities by description. Knowledge-Bart Based Systems, 2 210:106529, 2020.
13

---

A Task-Oriented Vulnerability Knowledge
In general, the task-oriented vulnerability knowledge employed by KF-GVD is derived from the matching rules of existing static analysis tools, relevant academic research findings, publicly available historical vulnerability information, and the expertise of security audit professionals. They are closely associated with the detected vulnerability types and contribute to the characterization of features relevant to the target tasks. Taking CWE-119 type vulnerabilities as an example, the fask-oriented vulnerability knowledge considered by KF-GVD includes program operations and sensitive functions related to the specific vulnerability type, as well as vulnerability knowledge associated with the specific detection target, as follows:
Vulnerable program operations: The triggering of vulnerabilities is often caused by certain typical risky behaviors in the code, According to our statistics, approximately 46% of CWE-119 vulnera- bilities in the Linux kernel are attributed to lack of boundary checks (as shown on the right side of Figure 2 Other contributing factors include the absence of input-output validation, resource consumption due memory recursion or iteration, and more.
Sensitive functions:
Sensitive functions are closely related to hazardous operations within a program. For CWE-119 vulnerabilities, we collect and predefine over 80 relevant C/C++ language sensitive functions, including string copy, concatenation functions (e.g., Strepy, strnepy, streat, strncat), input functions (e.g.. gets, scanf, fscanf), dynamic memory allocation functions (e.g., alloca) and on.
Customized knowledge for specific tasks: The manifestation of the same vulnerability type varies across different functional modules. For example, considering the two most frequently affected modules, fs and net, as illustrated in Figure vulnerabilities of CWE-119 type are most prevalent in code statements related to path parsing, symbolic links, and file access within the file system module. In the network module, key areas introducing CWE-119 vulnerabilities include network protocol stack parsing, network data processing, and on. As result, based on the collected historical vulnerability information and considering the functional characteristics of the target tasks, task-related vulnerability nodes can be marked.
B KF-GVD Model
The computation for each layer of the KF-GVD model can be denoted as follows:
GCN layer: After graph embedding, the model input is g = A), where X is the node feature matrix corresponding to G and the adjacency matrix corresponding to graph G. SAGPool uses a two-layer GCN structure for graph message propagation:
Hv Layers__y)ss A)))
Where o(.) is the activation function, Hy e Raxd where d is the hidden layer dimension of the network and n is the number of nodes in the graph G.
Self-attention layer: The self-attention layer calculates each node by of g based on the single-layer self-attention mechanism of GCN. The attention score of j E 1,.., n during message propagation, calculated as:
Sgy a OLh, -8 j=1
Sy e R is the final attention score matrix of nodes in g. where L == D AD 3, the Laplacian matrix obtained by normalizing a self connected adjacency matrix Á = A + I, D E Raxx is the degree matrix of A, lat e Rdx1, In this step, the layer will be based on S, preserves pon nodes with high weights and masks the hidden states corresponding to the remaining nodes, represented as:
idx = Top(Sg, v), Hv = mask Hv)
(5)
Where V = fo' (0mp)7m E V is the set of nodes with high weights retained in this layer, and is the retain ratio.
Graph pooling layer: Then, the graph pooling layer aggregates node features to form a fixed size representation, and the pooling result is based the features and topology of the graph. SAGPool uses average maximum pooling to obtain the global graph representation vector rg of g.
Tg Aug (h ))p=)) || Max (ther, )pm)))
(6)
14

---

Where he, E Hy is concatenation.
Classification layer: Finally, for the label prediction result yi for g is obtained from the fully connected layer and Softmax, and the expression is as follows:
Ds Softmar (MLP(Tg))
where ys E Y.
C Experiment Settings
C.1 Data Collection
To ensure that the model learns the potential vulnerability patterns implied in the target task as comprehensively as possible and achieves good generalization performance on the source task, the scale of the source task dataset must larger than that of the sub-task. This aligns with the current research and application scenarios.
All real data in our datasets are sourced from the National Vulnerabilityy Database (NVD), and collect information about target projects or vulnerability types based on publicly available security commit information related vulnerabilities. Positive and negative examples obtained based the location of the patch before and after program fixes. Moreover, we extract statement- labels based on the diff information the patches, identifying changes between new and old versions the statement level within functions.
During data processing, each CPG generated by Joern corresponds to function in the source file, and each graph association file associated with a source file enables mapping from node IDs in the CPG to source code statements. The experiment employs these preprocessed graph data directly as objects of study. Statistics on the source and target task datasets are provided in Table 6
Table 6: Dataset statistics
Dataset
Code Files
Vul
Non- Vul
Label Granularity
S119
6420
1:1.1
Function
S416
3631
1:1.5
Function
Tm
1122
1:2.5
Function, statement
Teub
460
1:1.1
Function, statement
C.2 Evaluation Metrics
The evaluation metrics used the experiment can be calculated as follows:
TP TP FP
(8)
TP
R TP+ FN
(9)
2x P
F1 = P+R
(10)
Here, TP refers True Positives, TN refers True Negatives, FP refers to False Positives, and FN refers to False Negatives. Precision measures the accuracy of the model in predicting positive instances, while Recall measures the coverage of the model among all actual positive instances. FI-Score is the harmonic mean of Precision and Recall.
For the evaluation metrics at the statement level, MAP can be calculated follows:
AP,
P(K) x rell
(11)
N
MAP
AP
(12)
Q
15

---

Where, K represents the top K statements with the highest vulnerability prediction probabilities, rel, denotes the number of actual vulnerable statements among the top K statements. N is the total number of samples, and Q is the number of classification categories.
C.3 Parameter Settings
Table 7: Parameter settings
Model
Parameter
Setting
Min count
0.001
Word2Vec
Size
30
Window
Embedding dim
300
Hidden dim
32
Activation funcion
Relu
GNNs
Learning rate
0.0001
Optimizer
Adam
Train: Validation: Test
8:1:1
During the feature extraction phase, CPGs corresponding source code files were generated using Joern version 1.1.1033. We employed a pre-trained Word2 Vec model for mapping the semantic information of CPG nodes (V seemantie) to features. The SAGPool model deployed in both source and target tasks were implemented using PyTorch version 1.4.0 and CUDA version 10.2. We conducted all experiments workstation equipped with a Quadro RTX 6000 GPU. The model parameters for KF-GVD are detailed in Table
D Supplementary Experiments
D.1 Vulnerability Knowledge Sensitivity Analysis
To investigate how the knowledge fusion approach of KF-GVD influences the VD performance on both source and target tasks, we analyze it from the following four aspects:
The ratio of samples implement knowledge fusion.
Fi
Fs
Drivers
25
(1-score(%)
Net
40
Includes
Includes
85
CWE-125
CWE-125
20
35
CWE-787
45
0.1
0.5
0.9
0.1
0.5
0.7
0.9
(a) Results on function-level VD
(b) Results statement-level VD
Figure 7: The F1-score of KF-GVD with knowledge fusion for different ratio of samples on Tmys and T sub.
We investigate how varying the ratio of fused samples a specific task affects the overall performance of KF-GVD. F shows shows the influence of the ratio of samples employed knowledge fusion (KF- ratio) on the Fl-score for function-level and statement-level VD on the target tasks T.m. and Trub- It can be observed that, for function- VD, an increase in the number of fused samples may influence the model's performance to some extent. Similarly, at the statement level, the performance tends to reach a "saturation" state with an increase in fused samples. The model achieves optimal results for both function-level and statement-level detection when the ratio samples subjected to the knowledge fusion operation is approximately 0.3-0.5 of the total task dataset. According to our statistics, the ratio of samples subjected to fusion in our practice ranges from 0.28-0.56, which aligns with the experimental expectations.
16

---

Knowledge subgraph,fuse coefficient.
100
Tm119
F1-score(50)
70
80
Tub
50
T Tm416
20
B
Figure 8: The average F1-score KF-GVD on different fusion coefficient settings.
In the experiments, we kept the feature values of the nodes the fusion coefficient to investigate the impact of vulnerability knowledge on model's feature learning for specific tasks. To ensure a model configuration with a certain level of consistency, we calculated the average Fl-score KF-GVD for all subtasks within each target task under different fusion coefficient settings. As shown in Figure 8 It can be observed that when the value of B is small, it has almost no impact on the overall performance of the model. The model performs well for specific tasks when beta is in the range of 1.5-2. However with the increase of value the influence of relevant knowledge on model feature learning is significant and the performance of the model decreases obviously and is almost ineffective. This also to some extent reflects that the manually introduced vulnerability knowledge adopted by KF-GVD, serving as a supplement to the current task conditions, cannot replace the inherent information in the source code for expressing vulnerability patterns.
(Equation I fixed and varied
Performance of the model fused with target task-related knowledge adopted the source tasks.
Table 8: Results of applying the target task model their corresponding source tasks.
(a) The results applying the models trained Tmnio to S416-
The results of applying the models trained on T Tmas6
and T.u to S119
Task
P(%)
R(%)
F(%)
Task
P(%)
R(%)
F(%)
S119
82.9
90.9
86.7
S416
86.8
89.3
88.0
Fs
91.7
91.1
91.4
Fs
91.9
85.0
88.3
Drivers
90.2
90.4
91.4
Drivers
89.7
88.4
89.1
Net
89.6
84.9
87.2
Net
81.82
94.7
87.8
Includes
93.7
86.4
89.9
Includes
79.3
98.8
88.0
CWB-125
77.1
89.8
82.9
Kernel
95.4
83.8
89.2
CWE-787
83.2
85.0
84.1
Block
86.6
82.4
84.4
AD
5.48
2.67
0.82
AD
-0.76
-0.5
-0.17
Table shows the results of applying the models trained with fused knowledge from target tasks to their corresponding source tasks. In the last row, calculate the Average Difference (AD) of evalua- tion metrics for the models before(in the first row) and after knowledge fusion for the corresponding source tasks. It can be observed that even though KF-GVD incorporates specific to certain tasks, still manages to maintain the initial VD performance to some extent on the source tasks, with an improvement on $119 and only marginal decreases within 1% for all metries on S416-
vulnerability knowledge
The impact of implementing different task-oriented vulnerability knowledge.
We conduct the ablation study on the impact of implementing different task-oriented vulnerability knowledge on the performance of KF-GVD. The knowledge related to specific vulnerability type (vulnerable program operations and sensitive functions) is denoted as KI, and the customized knowledge for specific tasks as K2. Table 9 shows F1 Fc-score (%) comparison of VD through the fusion of different vulnerability knowledge on the target tasks corresponding to S For the target
task T Tmigs using only KI resulted in an average increase of 11.9% in the Fl-score, and an average
17

---

improvement of 16.3% by K2 only; for the target task sus. KI resulted in an average increase of 15.1% the Fl-score, and K2 resulted in an average improvement of 7.5%. It can be observed that target tasks like Suu are more sensitive to K1. Furthermore, the combined integration of K and K2 resulted average F1 gains of 23.3% for mmno and 15.7% for T sub, further demonstrating the effectiveness of the proposed method.
Table Fl-score (%) comparison of VD through the fusion of different vulnerability knowledge on the target tasks corresponding to S119-
Strategy
aub
Fs
Drivers
Net
Include
CWE-125
CWE-787
GVD GVD KI
65.2
76.2
64.7
59.3
54.6
64.0
80.4
79.1
82.9
70.6
71.3
77.5
GVD + K2
88.6
85.9
78.6
77.4
62.6
71.0
GVD KI + K2
95.7
92.3
82.5
88.0
67.9
82.1
D.2
Feature Representation Sensitivity Analysis
We evaluate the impact of the feature representation method employed by KF-GVD on VD per- formance by using different graph representation methods and embedding models S119 and its corresponding target tasks.
Graph representation based on different code properties.
Table 10: F1-score (%) comparison of VI) using different graph representations S119 and its corresponding target tasks.
T m118
T
Graph
S119
Fs
Drivers
Net
Include
CWE-125
CWE-787
CFG
54.4
46.8
50.8
45.2
49.3
54.6
50.3
PDG
61.8
60.7
57.9
52.3
56.8
59.6
56.5
CPG
86.7
95.7
92.3
82.5
88.0
67.9
82.1
As shown Table compared to a single code property, the CPG-based code representation method achieves average F1 Fl-score improvement 27.1 % on target tasks. To enable the pre-trained general model to more flexibly adapt to a diverse range of downstream target tasks for enhanced VD, is crucial consider more comprehensive vulnerability features during the modeling stage. Although the CPG-based approach may result in slower performance compared to using a single code property during model training, we believe that this trade-off is justified and acceptable, given the potential gains in detection accuracy and model adaptability.
Generate node semantic features V Vemantiic using different embedding models.
Table 11: F1-score (%) comparison of VD using different embedding models on S119 and its corresponding target tasks.
T Tmn
Toub
Model
S119
Fs
Drivers
Net
Include
CWE-125
CWE-787
Vode2Vec
84.2
92.8
94.6
83.8
88.9
68.3
79.6
Code2Seq
67.5
70,9
68.4
63.3
66.1
52,8
61.4
Word2 86.7
95.7
92.3
82.5
88.0
67.9
82.1
Table IT shows the F1-score (%) comparison of VD using different embedding models on S119 and its corresponding target tasks. It can observed that the impact using Code2Vec and Word2Vec embedding methods on the performance of KF-GVD is very close, and both are superior to the Code2Seq embedding model. However, KF-GVD adopts Word2Vec as the embedding model for generating node semantic feature vectors for the following reasons: First, since the CPG already
18

---

incorporates AST properties, we consider the AST parsing process in Vede2Vec to be redundant and more time-consuming, whereas Werd2Vec is simpler and more efficient comparison. Second, while Code2Seq leverages attention mechanisms that provide advantages in handling complex code structures and long-distance dependencies. For KF-GVD, the complex structure and dependencies of the source code are represented by CPG as whole, and the source code corresponding to the CPG node a concise code snippet or statement generated by joern parsing. We infer that this also the reason for the relatively poor performance of the Code2Seq model.
D.3 The Performance of KF-GVD on Cross-Domain Tasks
Table 12: Fl-score (%)
comparison of VD on cross-domain tasks.
Method
L->F
L->0
F-SL
F-50
O->L
O-sF
Cppcheck
37.5
28.2
40.1
27.6
32.4
19.0
Flawfinder
27.5
18.6
22.8
24.1
14.8
25.9
Sysver
39.9
29.4
30.1
34.2
32.4
33.5
VulCNN
48.4
32.3
42.3
36.4
35.9
27.3
Devign
44.6
35.1
41.2
43.1
34.3
47.5
ReGVD
54.3
33.9
38.6
40.1
44.8
50.2
IVDetect
50.1
47.3
48.5
49.7
42.6
48.6
GVD-ft
55.8
50.0
53.5
48.0
62.1
54.7
KF-GVD
57.1
54.0
62.4
56.8
65.6
60.3
We further verify the generalization of KF-GVD in cross-domain VD scenarios, and conduct cross- domain VD among Linux kernel (L), FFmpeg (F) and Openssl (O). Table [2] shows the comparison of F1 scores (%) on six cross-domain tasks. It can be observed that compared to suboptimal methods, KF-GVD achieves an improvement of 1.3%-8.9% in F1-score, with an average gain 5.4%. The above experiments demonstrate the effectiveness of KF-GVD in cross-domain VD tasks compared to SOTAs. On the other hand, compared with specific CWE-oriented VD tasks (an average gain of 22.6% in Fl-score), KF-GVD performs relatively poorly in cross-domain tasks. The unknown vulnerability patterns between cross-domain projects, and the significant differences in programming paradigms and code functionality across projects pose challenges to effective knowledge transfer.
19

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

The method proposed in this paper is application-oriented and does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: In Section 4.1 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: The dataset has been uploaded to the supplementary materials, and the detail can be found in Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https w
//nips. //nips.cc/pil wwwppbllccgguueiiocccce
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
In section and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrp.ccccaaaaa/a//aaaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Section 1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We adopt own datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "DgiuH8VHPV", "reviewer": "Reviewer_Yw62", "summary": "This paper introduces KF-GVD, a knowledge fusion-based vulnerability detection method integrating specific vulnerability knowledge into the Graph Neural Network (GNN) feature learning process. Traditional VD methods apply uniform feature learning, which can miss diverse vulnerability types or functional modules. KF-GVD integrates specific vulnerability knowledge into the feature learning process, enhancing accuracy across different modules and subtypes without sacrificing general performance.", "questions": "See the weakness section.", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Novelty: Novel technique to combine the generalizability of the pre-trained knowledge of vulnerability detection along with Task-Specific knowledge.\n\n-Task-Specific Knowledge Integration: Integrating task-specific knowledge makes the system more specialized on the domain.\n\n-Comprehensive Methodology: Methodology is presented comprehensively.", "weaknesses": "-More Statistics (only precision and recall is given) such as confidence intervals or p-values. Including these would provide a more robust evidence base for the claims and enhance the credibility of the results.\n\n-Broader Dataset Evaluation (The evaluation is primarily conducted on the Linux kernel and a few C/C++ open-source projects.\n\n-More discussion is needed in limitations and future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed review and valuable suggestions. We have taken your comments into consideration and respond to the raised points below.\n\n**1) P-values Evaluation**\n\nWe evaluated the improvements obtained by using the model fine-tuning strategy (GVD-ft) and the vulnerability detection method based on knowledge fusion (KF-GVD) in different target task scenarios corresponding to S119 and S416, respectively, compared with the same general task model. The following table shows the p-values of these two main improvement methods on each dataset. Meanwhile, more detailed p-value statistics for other comparative experiments will be promptly updated in the paper.\n\n|$S_{119}$|Fs|Drivers|Net|Include|CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|\n| GVD-ft | 0.058 | 0.062 | 0.046 | 0.051 | 0.088 | 0.079 | \n| KF-GVD |0.0036 | 0.0048| 0.0032 | 0.0043 | 0.0087 | 0.006 |\n\n|  $S_{416}$  | Net | Fs | Drivers | Kernel | Block | Include |\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|\n| GVD-ft | 0.077 | 0.042 | 0.13| 0.071 | 0.094 |0.055| \n| KF-GVD | 0.0028 | 0.0035| 0.002| 0.0047| 0.0022 | 0.0029|\n\nIt can be observed that the improvements brought about by using the task-oriented knowledge fusion method proposed in this article provide detection results of significant value. KF-GVD achieves significantly better detection performance on the target datasets corresponding to S119 and S416 compared to the baseline model.\n\n**2) Broader Dataset Evaluation**\n\nWith the continuous improvement of vulnerability data and the demand for practical application scenarios, we will continue to evaluate KF-GVD on a wider range of software projects and vulnerability type datasets, which will help to continuously improve and verify the effectiveness of the method to serve practical applications. In this paper, we choose to evaluate KF-GVD mainly on CWE-119 and CWE-416 and the Linux kernel modules for the following reasons: First, the two types of vulnerabilities and their subtypes, CWE-125 and CWE-787, are included in the 2023 CWE Top 25 Most Dangerous Software Weaknesses. Since KF-GVD is proposed to be application-oriented, it is important to focus on these more prevalent types of vulnerabilities. Additionally, linux kernel and projects such as ffmpeg and openssl used in Appendix E.2 are one of the most common vulnerability detection targets in in this field of research. Therefore, evaluating KF-GVD on these projects provides a more intuitive comparison. Furthermore, for the evaluation objects employed in our paper, we currently have relatively more complete data sets to ensure the objectivity and validity of the experiments. We will also make this data publicly available in hopes of aiding research in this field.\n\n**3) Limitations and Future Work**\n\nIn the data processing stage, the time spent generating CPG data from source files accounts for more than 75% of the total processing time. More efficient data generation tools and processing strategies should be considered in the future. During the feature embedding stage, we truncated the $V_{code}$ that exceeded the feature vector length threshold, which led to a certain degree of semantic information loss. Furthermore, although KF-GVD has achieved significant improvements in statement-level vulnerability localization compared to SOTAs, there is still considerable room for improvement in fine-grained localization performance compared to function-level detection results. In the next phase, combining the powerful representation and generation capabilities of current large code models to obtain more comprehensive and efficient source code features will be a future research direction to further improve the performance of vulnerability localization. Finally, since the representation object of CPG is a function, KF-GVD cannot detect vulnerabilities that have cross-function or cross-file dependencies. Considering more comprehensive code call relationship extraction schemes in the future will help efficiently discover more complex and varied types of vulnerabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XziAY7OSXj", "reviewer": "Reviewer_G6qq", "summary": "This paper proposes KF-GVD, a Graph Neural Network (GNN) model that integrates specific vulnerability knowledge into its feature learning process to enhance vulnerability detection accuracy in source code. Unlike traditional deep learning methods that optimize for general performance, KF-GVD uses knowledge fusion to tailor its detection capabilities to specific tasks, effectively addressing diverse functional modules and vulnerability subtypes. Extensive experiments demonstrate KF-GVD's superiority over state-of-the-art methods, with notable improvements in precision and recall. Additionally, KF-GVD discovered nine previously undisclosed vulnerabilities in open-source C/C++ projects, showcasing its practical effectiveness in real-world applications.", "questions": "1. Is it possible some Sota static analysis tools could also detect such vulnerabilities? For example, why don't you try to write CodeQL rules to detect vulnerabilities for corresponding CWE types, or can such tools not be applied in your settings?\n    \n2. Have you considered evaluating KF-GVD on a wider range of software projects and vulnerability types beyond CWE-119 and CWE-416 and the Linux kernel modules?\n        \n3. If I understand right, according to section 3.2.1 the CPG is constructed based on function level, so how do you consider the vulnerabilities that have interprocedural issues? Do you need to construct an iCFG(Maybe iCPG here) to let GCN learn its features?  And why don’t you use CFG to capture the features? Will it have worse results than current CPG-based methods? In most cases, a Control Flow  Graph may be enough. CPG combines more features and may be slower than CFG, but which would have better results is not sure.", "limitations": "The paper does acknowledge some limitations and challenges, particularly in the context of applying the proposed KF-GVD model, but leave this in appendix, I think this should be mentioned in main paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n1. The paper presents a novel approach to vulnerability detection by integrating task-oriented knowledge into a GNN model.\n    \n2. The results are robust, showing significant improvements in precision and recall across various datasets and tasks. The paper includes detailed explanations of the methodology, thorough evaluations, and a case study that validates the practical effectiveness of KF-GVD in real-world applications. I love the case studies.\n    \n3. The paper is a great presentation.\n    \n4. This paper shows its potential to substantially improve vulnerability detection in software development. By tailoring the detection process to specific tasks and contexts, KF-GVD addresses a critical gap in existing methods, which often fail to capture the nuanced patterns of different vulnerability types and functional modules.\n    \n5. The result comparison is comprehensive, including static analysis tools and LLM-assisted tools.", "weaknesses": "Weakness:\n\n1. KF-GVD relies heavily on the quality and relevance of the task-specific vulnerability knowledge integrated into the model. The process of defining and extracting this knowledge is not extensively discussed, and the paper does not address potential challenges or limitations in obtaining accurate and comprehensive knowledge. More details on how this knowledge is curated, validated, and updated over time would be beneficial.\n    \n2. The limitation section is in the appendix; I think it should be moved to the main content.\n    \n3. While the paper mentions that KF-GVD achieves high transparency and interpretability through self-attention mechanisms, it does not provide sufficient details on how these aspects are practically implemented and evaluated.\n    \n4. Could you please provide pdf-editable figures?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our paper , which is very helpful to us. Below are the answers to the questions you have raised, hoping to solve your doubts.\n# Answer for Q.1\nSome Sota static analysis tools may detect such vulnerabilities. In our comparative experiments, we also considered rule-based commercial static code analysis tools such as Cppcheck and Flawfinder. Based on the implementation principles of these rule-based static analysis tools, they can be applied to most static vulnerability detection scenarios. However, these rule-based methods have the following disadvantages: First, they rely on the pre-definition of vulnerability rules by developers and need to be continuously updated and maintained to ensure the effectiveness and completeness of the rules. In addition, when limited static pattern matching rules are applied to complex and diverse detection targets, it results in a high false positive and false negative rate. In contrast, the DL-based VD method can automatically learn vulnerability feature patterns without relying on manually written matching rules. Moreover, the task-oriented knowledge fusion method proposed in our paper can realize customized update and application of existing vulnerability knowledge with lower maintenance cost.\n# Answer for Q.2\nWith the continuous expansion of vulnerability data and the demand for practical application scenarios, we will continue to evaluate KF-GVD on a wider range of software projects and vulnerability types, which will help to continuously improve and verify the effectiveness of the method to serve practical applications. In addition, we are pleased to share that in a recent research project, KF-GVD was applied to vulnerability detection in BusyBox version 1.36.1 and discovered and confirmed two undisclosed vulnerabilities. Thus, a more comprehensive evaluation in the future will be beneficial and necessary. \n# Answer for Q.3\nCurrently, the CPG generated by Joern for KF-GVD corresponds to individual functions. Therefore, as mentioned in the Threats to Validity section in Appendix F, the method does not consider vulnerabilities that have interprocedural issues. Your comments are very valuable, which will guide the further improvement and optimization of the method in our future research.\n\nWe use CPG instead of CFG alone because CPG can model most types of vulnerabilities, which has been proven by the proposer of Joern. For KF-GVD, in order to make the pre-trained general task model more flexibly adapt to a wider range of downstream target tasks to achieve more efficient vulnerability detection, it is critical to consider more comprehensive vulnerability features in the modeling stage. The following table shows the F1-score (%) comparison of vulnerability detection using CFG, PDG, and CPG for vulnerability feature extraction on the target tasks corresponding to S119 and S416. \n\n||$S_{119}$|Fs|Drivers|Net|Include|CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|CFG|54.4|46.8|50.8|45.2|49.3|54.6|50.3| \n|PDG| 61.8 | 60.7 | 57.9 | 52.3| 56.8 |59.6|56.5|\n|CPG| 86.7 | 95.7 | 92.3|  82.5 | 88.0 |67.9|82.1|\n\n||$S_{416}$|Net|Fs|Drivers|Kernel|Block|Include|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|CFG| 42.9|38.1|42.7|36.3|44.8|30.2|32.7| \n|PDG|69.6|66.2|67.9| 57.6| 62.4|59.9|64.3|\n|CPG|88.0|87.3|82.9|78.7|88.6|74.1| 87.5 |\n\nIt can be observed that the CPG-based code representation can achieve the best vulnerability detection results on different target tasks compared with a single code property.  While it is true that the CPG-based method will be slower than CFG during the model training stage, we consider that this trade-off is justified and acceptable. Specifically, in our experiments, the CPG-based method takes an average of 11.5 seconds to analyze a sample, whereas using CFG alone takes about 9.6 seconds.\n# Answer for Weakness\n1) In general, task-specific vulnerability knowledge comes from the matching rules of existing static analysis tools, relevant academic research results, and public historical vulnerability information, most of which are publicly available and easily accessible. In the majority of application scenarios, historical vulnerability information related to the target software is one of the primary and direct sources of vulnerability knowledge. The maintenance of these vulnerability data depends on the vulnerability database and software developers. In the process of realizing task-oriented knowledge fusion, the retrieval, collection, and data preprocessing of this information will be automated and efficient. In addition, the continuous expansion of the data of the various vulnerability knowledge will also contribute to continuously optimize the generalization of the source task model.\n2) Thank you very much for your suggestions. Due to page limitations at this stage, we have placed the limitations in the appendix. The subsequent version will be adjusted, and the current figures will be updated to PDF-editable figures.\n3) The high transparency and interpretability of KF-GVD are achieved through the model self-attention mechanism, with the aim of achieving fine-grained vulnerability localization. The attention weights given to different graph nodes in each instance (CPG) by the graph neural network model reflect the importance of nodes that affect the current model decision (vulnerable code/benign code). On this basis, we consider the code statements associated with these nodes as potential statements introducing vulnerabilities for fine-grained vulnerability localization. In the experiments, the MAP metric is used to evaluate the average precision of the top K confidence predictions of the model, specifically assessing whether the associated code of the nodes with the top K highest attention scores are real vulnerability lines, thus evaluating the model's fine-grained vulnerability localization performance and the rationality of the decision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGPET1jXSZ", "reviewer": "Reviewer_YqLp", "summary": "The paper proposes KF-GVD, a vulnerability detection method that integrates task-specific vulnerability knowledge into a graph neural network model. KF-GVD aims to guide the model to learn vulnerability patterns tailored to the target task, rather than relying solely on a generalized approach. Experiments show KF-GVD outperforms baseline methods, especially on tasks involving specific code modules or vulnerability subtypes. KF-GVD also provides improved interpretability by identifying the most influential source code statements related to predicted vulnerabilities. While KF-GVD demonstrates the benefits of incorporating task-specific knowledge, the paper acknowledges limitations such as its dependence on the availability of relevant knowledge and weaker performance on cross-domain tasks compared to the gains on target-specific tasks.", "questions": "1. How does the KF-GVD approach differ from the \"Interpreters for GNN-Based Vulnerability Detection\" paper in terms of its core technical contributions and unique aspects? Please provide a more detailed analysis to clearly differentiate the novelty of your work.\n\n2. Could you please conduct a more rigorous comparative evaluation between KF-GVD and other state-of-the-art GNN-based vulnerability detection methods, including the \"Interpreters for GNN-Based Vulnerability Detection\" paper? This would help better demonstrate the clear advantages of the KF-GVD approach across multiple performance metrics and target tasks.\n\n3. What was the rationale behind choosing the Word2Vec embedding technique over code-specific embedding methods, such as code2vec or code2seq? Could you provide a comparative analysis of the performance using different code embedding techniques to justify the choice made in the KF-GVD framework?\n\n4. To enhance the clarity and organization of the paper, could you restructure and simplify the technical descriptions to improve the overall accessibility? Could you also consider adding visual aids, such as diagrams or illustrations, to better explain the key components and workflow of the KF-GVD framework?", "limitations": "1. The paper does not provide a thorough discussion of the computational efficiency and scalability of the KF-GVD approach. As the size and complexity of software projects increase, the computational requirements and the ability of the method to handle large-scale code bases should be addressed.\n\n2. The paper does not explore the potential limitations of the code embedding technique used (Word2Vec) and how it may impact the overall performance of the vulnerability detection. A more comprehensive analysis comparing different code embedding methods would strengthen the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Quality:\nThe technical implementation of the KF-GVD framework appears to be reasonably well-designed, with a solid GNN architecture and a knowledge fusion process. The extensive experiments across multiple target tasks, including function-level and statement-level evaluation, demonstrate a thorough empirical assessment of the approach. The authors' attention to interpretability through the self-attention mechanism is a positive aspect.\nClarity:\nThe paper is technically dense but reasonably well-written. The authors provide clear descriptions of the key components of the KF-GVD framework, including the feature representation, knowledge extraction, and vulnerability detection and interpretation. The experimental setup and results are presented in a structured and comprehensible manner.\nSignificance:\nImproving the effectiveness of vulnerability detection, especially for specialized tasks and target code, remains an important and relevant problem in software security. The KF-GVD paper's exploration of leveraging task-specific knowledge for vulnerability detection could contribute to ongoing efforts in this area, even if the core ideas are not entirely novel.", "weaknesses": "Lack of Novelty:\nThe core idea of integrating task-specific vulnerability knowledge into a GNN-based detection model is not novel, as the \"Interpreters for GNN-Based Vulnerability Detection\" paper has already explored a similar approach. The authors' failure to identify and properly differentiate their work from this closely related prior research is a major weakness.\nInsufficient Literature Review:\nThe paper lacks a thorough review of the existing literature on vulnerability detection, particularly in the context of GNN-based techniques and the use of task-specific knowledge. The authors should have conducted a more comprehensive survey to situate their work within the broader research landscape and identify the specific contributions they are making.\nUnclear Uniqueness of Contributions:\nWithout a clear and compelling unique contribution, the overall significance and impact of the KF-GVD paper are diminished. The authors need to re-evaluate the paper's focus and either identify a novel aspect of their approach or reposition the work to highlight its incremental advances over prior research.\nPotential Overlap with Existing Work:\nThe significant similarities between the KF-GVD paper and the \"Interpreters for GNN-Based Vulnerability Detection\" paper raise concerns about potential overlap or duplication of efforts. The authors should carefully examine the extent of this overlap and ensure that their work is not merely reiterating or incrementally building upon existing research.\nLack of Rigorous Comparative Analysis:\nThe paper does not provide a rigorous comparative analysis between KF-GVD and other closely related GNN-based vulnerability detection methods, particularly the \"Interpreters for GNN-Based Vulnerability Detection\" paper. A more comprehensive and systematic comparison would be necessary to demonstrate the unique merits and contributions of the KF-GVD approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments. We have carefully considered the issues you raised and provided our responses below, hoping to address your concerns.\n# Answer for Q.1\nBased on the research content of the paper \"Interpreters for GNN-Based Vulnerability Detection\" (hereafter referred to as \"that paper\" for brevity), we believe that the work of KF-GVD does not overlap with it. We analyzed and compared the two papers from the following aspects:\n\n**1) Research Objectives:** That paper aims to evaluate the capability of different GNN interpreters to explain VD models. Our research is application-oriented, aiming to adopt the most suitable vulnerability pattern learning and identification strategies in VD for different tasks, so that the model can achieve more efficient and scalable VD on a variety of target objects.\n\n**2) Contributions:** That paper is the first empirical study to evaluate the ability of GNN interpreters on VD model. The study proposes principled guidelines to assess the quality of the interpretation approaches for GNN-based vulnerability detectors; Our research proposes a VD framework KF-GVD, which is the first task-oriented GNN-based VD model. KF-GVD adopts a simple and efficient task-oriented knowledge fusion method to enable the general-purpose model can be flexibly adapted to VD on various target tasks in practical applications while maintaining the generalization performance for source task.\n\n**3) Evaluation:** That paper evaluates the quality of interpretations of different GNN interpreters on VD models based on the proposed stability, robustness and effectiveness metrics; KF-GVD evaluates the VD performance compared with SOTAs based on precision, recall, F1-score and MAP.\n\n**4) Interpretability:** In that paper, the study introduces GNN interpreters to achieve the interpretation of VD models. For KF-GVD, the interpretability of our method comes from the graph self-attention mechanism adopted by the model rather than the introduction of a special interpreter, which is more efficient and intuitive on large-scale data. Furthermore, KF-GVD achieve fine-grained vulnerability localization by leveraging the inherent interpretability of the model to better serve practical applications. In other words, interpretability is the nature of KF-GVD rather than its purpose.\n\n# Answer for Q.2\nAccording to the three evaluation criteria proposed in that paper, we applied the three GNN interpreters discussed in that paper to explain KF-GVD and compared these methods with the self-attention mechanism (Self-Att) adopted by KF-GVD. Due to word count limitations, the following provides partial evaluation results and concise analysis on the source task S119 and its corresponding target task dataset. \n\n**1) Effectiveness:** \n||S119|Fs |Drivers| CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|\n|GradCAM|48.6|41.5|50.4|39.7|39.5| \n|GNNExplainer|33.9|32.1|33.7|30.6|32.4|\n|DeepLIFT|44.2|35.4|47.6|36.5|38.9|\n|Self-Att|54.4|42.7|51.8|38.9|44.8|\n\nThe effectiveness evaluation criteria IoU (%) can reflect the accuracy of the association between the interpreter results and the ground-truth. It can be seen that for KF-GVD, the vulnerability localization based on the self-attention mechanism is the most effective compared with other interpretation methods.\n\n**2) Stability:** \n||S119 |Fs|Drivers|CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|\n|GradCAM|36.9|37.3|38.5|41.8|35.5| \n|GNNExplainer|68.7|69.2|66.1|64.0|68.9|\n|DeepLIFT|5.6|3.3|1.2|4.7|2.2|\n|Self-Att|57.2|54.6|33.8|34.5|37.9|\n\nIt can be observed that adjustments to the model parameters impact the stability of the self-attention mechanism directly related to the model. However, compared to the decomposition-based interpretability method DeepLIFT, the results are still relatively better.\n\n**3) Robustness:** \n||S119|Fs |Drivers| CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|\n|GradCAM|43.7|41.0|43.3|38.6|42.5| \n|GNNExplainer|13.6|14.3|12.1|11.8|12.4 |\n|DeepLIFT|44.9|37.6|42.3|40.7|39.2|\n|Self-Att|62.4|58.9|60.9|49.7|43.1|\n\nThe robustness criteria can reflect the quality of the interpreter's explanation on some examples related to code cloning. As part of the model structure and benefiting from the adaptability of KF-GVD to different tasks, the self-attention mechanism has better robustness than GradCAM and DeepLIFT.\n\n# Answer for Q.3\nThe following table shows the F1-score (%) comparison of vulnerability detection using different code embedding techniques on S119 and its corresponding target task datasets.\n||S119|Fs |Drivers|Net|Include|CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Code2Vec|84.2|92.8|94.6|83.8|88.9|68.3|79.6| \n|Code2Seq|67.5|70.9|68.4|63.3|66.1|52.8|61.4|\n|Word2Vec|86.7|95.7|92.3|82.5|88.0|67.9|82.1|\n\nIt can be observed that the impact of using Code2Vec and Word2Vec embedding methods on the performance of KF-GVD is very close, and both are superior to the Code2Seq embedding method. KF-GVD chooses the Word2Vec embedding method for the following reasons: Code2Vec maps code snippets into vector space through path embeddings based on the AST during the encoding process. However, since the CPG already contains the AST property, the source code corresponding to each node is a code snippet obtained based on CPG parsing. Therefore, the AST parsing process of Code2Vec is redundant, and it will also incurs more time consumption at the embedding stage. In contrast, Word2Vec is simpler and more efficient. \n\nThe Code2Seq embedding method combines the attention mechanism and has advantages in processing complex code structures and long-distance dependencies. However, in the feature embedding stage of KF-GVD, the complex structure and dependencies of source code files are represented as a whole through the CPG. The source code corresponding to CPG nodes are usually very short code snippets. This is why the Code2Seq embedding method performs relatively poorly on KF-GVD.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6ERZJOwe9", "reviewer": "Reviewer_78X5", "summary": "This paper proposes KF-GVD, a novel Knowledge Fusion-based Graph Neural Network (GNN) model designed for detecting vulnerabilities in C/C++ source code. KF-GVD employs Code Property Graphs (CPGs) to represent the structure and semantics of the source code. Based on the CPGs, KF-GVD flexibly extracts features and tailors them for specific tasks. Through this innovative approach, KF-GVD is able to fine-tune its detection mechanisms. The experimental results demonstrate that KF-GVD is effective in Vulnerability Detection (VD), by outperforming the state-of-the-art static analysis- and ML-based VD methods. Notably, KF-GVD successfully identified nine 0-day vulnerabilities, indicating its potential impact on software security.", "questions": "## Q.1 What is the motivation behind using CPG to represent vulnerability semantics?\n\nWhile KF-GVD demonstrates the impressive performance in vulnerability detection (VD), it still remains unclear why the authors employ CPG instead of the alternatives such as CFG. From my understanding, the Code Property Graph (CPG) can be regarded as a fusion of Abstract Syntax Trees (AST), Control Flow Graphs (CFG), Call Graphs (CG), and Program Dependence Graphs (PDG). Although it is undoubtedly powerful, building CPG for large-scale projects like Linux Kernel could be extremely expensive, particularly in terms of computing program dependence. \n\n- **Q.1.1) for authors to respond:** During the feature extraction process, have there been any instances where constructing the CPG failed? If so, how do the authors manage these construction failures?\n\n- **Q.1.2) for authors to respond:** Could it be possible to use CFG or other alternatives to replace CPG? Can the authors justify the superiority of using CPG on this task?\n\nAdditionally, I'm curious about the decision not to use LLVM Intermediate Representation (IR) to analyze the C/C++ projects in KF-GVD. Typically, the workflow for analyzing C/C++ projects involves the following steps;\n\n    - Compile the source code into a kind of IR (not limited to LLVM IR).\n    - Build corresponding graph-based structures (e.g., CFG or PDG) according to the requirements.\n    - Pass them to downstream applications (could be either static analyzer or neural network). \n\nSince LLVM IR offers a rich, standardized structure that provides detailed information on control and data flow, I  believe using it can potentially enhance the capability of KF-GVD. Moreover, employing LLVM IR instead of solely depending on the source code (in the representation of AST) can minimize the noise. For instance, macros are expanded and machine-dependent code is removed. \n\n- **Q.1.3) for authors to respond:** Can the authors justify the reason for excluding LLVM IR?\n\n## Q.2 How are the code statements determined to be related to vulnerabilities?\n\nI find the explanation of the statement-level vulnerability detection (VD) in the paper to be somewhat ambiguous. For clarity, could the authors explain how to determine which lines of code are marked as vulnerable?", "limitations": "- The size of the dataset used in the study is relatively small, which could impact the generalizability of the KF-GVD model. However, I also understand the reality that it is hard to obtain high-quality and open-source vulnerability datasets.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written, and I enjoy the reading.\n\n- The authors conducted a rigorous and comprehensive comparison of KF-GVD against the state-of-the-art, spanning from traditional static analysis tools to recently emerging machine learning-based approaches. \n\n- KF-GVD significantly outperforms SOTAs, achieves notable improvement in vulnerability detection, and exposes nine 0-day vulnerabilities.", "weaknesses": "- The paper does not sufficiently justify the use of CPG to represent vulnerability semantics, neglecting to discuss the potential alternatives such as Control Flow Graph (CFG).\n\n- The paper misses details on feature extractions. \n\n- The description for statement level VD is ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the detailed review and the professional opinions you have provided, which are highly valuable to us. Below are our responses to the questions you raised, and we hope they address your concerns.\n# Answer for Q.1\n+ **A.1.1)** In most cases, Joern can successfully build CPG. During the data processing phase, we found that some functions could not generate corresponding CPGs normally. According to our observations, these instances are usually related to missing compilation dependencies, complex macro definitions, nested preprocessor directives, and so on. However, the proportion of such cases in the target files involved at this stage is very small, so we usually ignore them.\n\n+ **A.1.2)** We use CPG instead of CFG alone because CPG can model most types of vulnerabilities, which has been proven by the proposer of Joern. For KF-GVD, in order to make the pre-trained general model more flexibly adapt to a wider range of downstream target tasks to achieve more efficient vulnerability detection, it is critical to consider more comprehensive vulnerability features in the modeling stage. The following table shows the F1-score (%) comparison of vulnerability detection using CFG, PDG, and CPG for vulnerability feature extraction on the target tasks corresponding to S119 and S416. \n\n||$S_{119}$|Fs|Drivers|Net|Include|CWE-125|CWE-787|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|CFG|54.4|46.8|50.8|45.2|49.3|54.6|50.3| \n|PDG| 61.8 | 60.7 | 57.9 | 52.3| 56.8 |59.6|56.5|\n|CPG| 86.7 | 95.7 | 92.3|  82.5 | 88.0 |67.9|82.1|\n\n||$S_{416}$|Net|Fs|Drivers|Kernel|Block|Include|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|CFG| 42.9|38.1|42.7|36.3|44.8|30.2|32.7| \n|PDG|69.6|66.2|67.9| 57.6| 62.4|59.9|64.3|\n|CPG|88.0|87.3|82.9|78.7|88.6|74.1| 87.5 |\n\nIt can be observed that the CPG-based code representation can achieve the best vulnerability detection results on different target tasks compared with a single code property.  While it is true that the CPG-based method will be slower than CFG during the model training stage, we consider that this trade-off is justified and acceptable. Specifically, in our experiments, the CPG-based method takes an average of 11.5 seconds to analyze a sample, whereas using CFG alone takes about 9.6 seconds.\n\n+ **A.1.3)** The reasons for KF-GVD not choosing LLVM IR are as follows:\n\n_Loss of High-Level Semantic Information_: LLVM IR's representation is closer to the machine code level and focuses on instruction-level operations. Such standardized representation is fine-grained and detailed, but to a certain extent, it can lead to a loss of high-level source code logic structure and semantic information, thereby affecting vulnerability analysis performance. Compared with the CPG generated by Joern, the high-level representation is typically more efficient in the vulnerability identification process.\n\n _Compilation Overhead:_ LLVM IR requires the source code to be compiled first, the compilation phase that is time-consuming and makes large-scale vulnerability detection difficult. VulDeeLocator, a fine-grained vulnerability detector based on LLVM IR, is a typical example. It takes about 30.7 seconds to analyze a single sample, which is about three times that of KF-GVD.\n\n_Tool Support for Security Applications:_ While LLVM IR is a powerful tool for compiler optimization and low-level code analysis, it lacks tool support for security applications; Joern, on the other hand, is a tool designed for security analysis and vulnerability mining, with better ease of use, convenient query and analysis functions and interfaces, which provides the possibility for subsequent implementation of fine-grained localization of code associated with graph nodes and efficient vulnerability data management.\n\n# Answer for Q.2\nKF-GVD achieves fine-grained localization of potential vulnerability statements based on the graph self-attention mechanism adopted in the model. The graph self-attention mechanism calculates an attention weight for each pair of neighbor nodes, reflecting the importance of a specific neighbor when aggregating information for a node. Through these attention weights, we can understand which neighbor nodes contribute more to the features of the current node, thereby providing a certain level of interpretability. \n\nSpecifically, for an instance (CPG), KF-GVD calculates and obtains the graph nodes with higher attention scores in the current graph based on the graph self-attention mechanism. Since the attention score reflects the degree of influence of the node on the current decision made by the model (vulnerable/benign code), when the sample is predicted as vulnerable, KF-GVD considers the source code statements associated with these high-score nodes as potential vulnerability statements. In practice, the graph association file generated by joern can realize the mapping from CPG nodes to source file code statements, thereby achieving fine-grained localization of vulnerability statements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary of the Paper**

This paper proposes KF-GVD, a knowledge fusion-based vulnerability detection method that integrates security vulnerability knowledge into a graph neural network feature learning process.

**Summary of Reviews**
- Reviewer Yw62 (Score 6 - Weak Accept): The reviewer commends the novelty, task-specific knowledge integration, and comprehensive methodology. They suggest additional statistics than precision and recall to provide more robust evidence for the authors' claims and to enhance the credibility of the results. They also suggest additional discussion on the limitations and future work.
- Reviewer G6qq (Score 6 - Weak Accept): The reviewer commends the novelty of integrating task-oriented knowledge into a GNN model, the robustness of the results across various datasets and tasks, and the great presentation. They also commend the comprehensive result comparison, including static analysis tools and LLM-assisted tools. They notice that the method relies heavily on the quality and relevance of the integrated task-specific vulnerability knowledge into the model. They also consider whether SOTA static analysis tools can detect the same vulnerabilities and whether the method should be evaluated on a wider range of software projects and vulnerability types.
- Reviewer YqLp (Score 4 - Borderline Reject): The reviewer commends the quality of the framework design and the extensive experiments performed across multiple tasks. They consider the lack of novelty, given that a recent paper has already explored the topic, and that the insufficient literature review does not adequately represent the uniqueness of the paper's contributions. 
- Reviewer 78X5 (Score 7 - Accept): The reviewer commends the well-written paper and the rigorous and comprehensive comparison of KF-GVD against the SOTA methods. They comment that the paper does not sufficiently justify the use of CPG to represent vulnerability semantics, misses details on feature extractions, and is ambiguous about statement-level vulnerability detection.

**Assessment**

Regarding Reviewer Yw62's concerns, the authors provide additional evaluations to compare the model fine-tuning strategy with their knowledge fusion-based vulnerability detection method, though they do not provide an in-depth explanation of these results. They do not provide additional evaluations on a wider range of software projects, stating that the vulnerabilities evaluated are among the most dangerous software weaknesses and that the target projects are among the most common vulnerability detection targets. They also recognize that data generation time accounts for the majority of the total processing time, that fine-grained localization can still be improved, and that KF-GVD cannot detect vulnerabilities with cross-function or cross-file dependencies.

Regarding Reviewer G6qq's concerns, the authors cite the disadvantages of rule-based static analysis tools, particularly their high false positive and false negative rates. They claim that their method comes with lower maintenance costs. They share that KF-GVD was able to discover and confirm two undisclosed vulnerabilities in an open-source project. 

Regarding Reviewer YqLp's concerns, the authors state that their work differs from past work as application-oriented, proposing a task-oriented GNN-based VD framework, rather than evaluating the performance of GNN interpreters in explaining VD models. They show that the self-attention mechanism adopted by KF-GVD outperforms other VD methods and justify their choice of code embedding.

Regarding Reviewer 78X5's concerns, the authors justify their choice of CPG to represent vulnerability semantics and explain that the fine-grained localization achieved by KF-GVD provides a level of interpretability. 

Overall, reviewers commend the novelty of KF-GVD, as well as the extensive experimental evidence provided to evaluate its effectiveness in vulnerability detection compared to SOTA methods. Additionally, the real-world impact of the method is clear from the authors' detection of 9 undisclosed vulnerabilities in C/C++ open-source projects. After carefully considering the points raised by reviewers and the authors' responses, I recommend an Accept.
Decision: Accept (poster)