DeepDFA: Dataflow Analysis-Guided Efficient Graph Learning for Vulnerability Detection
Authors: Benjamin Steenhoek
Wei Le
Hongyang Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
vulnerability detection
dataflow analysis
program analysis

Abstract: Deep learning-based vulnerability detection models have recently been shown to be effective and, in some cases, outperform static analysis tools. However, the highest-performing approaches use token-based transformer models, which do not leverage domain knowledge. Classical program analysis techniques such as dataflow analysis can detect many types of bugs and are the most commonly used methods in practice. Motivated by the causal relationship between bugs and dataflow analysis, we present DeepDFA, a dataflow analysis-guided graph learning framework and embedding that use program semantic features for vulnerability detection. We show that DeepDFA is performant and efficient. DeepDFA ranked first in recall, first in generalizing over unseen projects, and second in F1 among all the state-of-the-art models we experimented with. It is also the smallest model in terms of the number of parameters, and was trained in 9 minutes, 69x faster than the highest-performing baseline. DeepDFA can be used with other models. By integrating LineVul and DeepDFA, we achieved the best vulnerability detection performance of 96.4 F1 score, 98.69 precision, and 94.22 recall.

Under review as a conference paper at ICLR 2023
DEEPDFA: DATAFLOW ANALYSIS-GUIDER EFFICIENT GRAPH LEARNING FOR VULNERABILITY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning-based vulnerability detection models have recently been shown to be effective and, in some cases, outperform static analysis tools. However, the highest-performing approaches use token-based transformer models, which do not leverage domain knowledge. Classical program analysis techniques such as dataflow analysis can detect many types of bugs and are the most commonly used methods in practice. Motivated by the causal relationship between bugs and dataflow analysis, we present DeepDFA, a dataflow analysis-guided graph learn- ing framework and embedding that use program semantic features for vulnera- bility detection. We show that DeepDFA is performant and efficient. DeepDFA ranked first in recall, first in generalizing over unseen projects, and second in FI among all the state-of-the thrt models we experimented with. It is also the small- est model in terms of the number of parameters, and was trained in 9 minutes, 69x faster than the highestt-prforminn models. By integrating Vul and DeepDFA, we achieved the best vulnerability detection performance of 96.4 Fl score, 98.69 precision, and 94.22 recall.
baseline. DeepDFA can be used with other
INTRODUCTION
Software vulnerabilities cause great harm to people and corporations. Many Internet users have had their personal information breached because of security vulnerabilities, with common reports of breaches exposing millions of records (wik (02T). The average data breach costs the target company $4.24 million, according to IBM's 2021 report (ibm 2021). The number of vulnerabilities is growing every year, as reported by the Common Vulnerability Enumeration (CVE) from 2016-2021 (eve) 2021]. Because of its importance, software companies invested heavily to develop vulnerability detection tools that can scan software before its release (Lu ct al. 202I Zheng et al. 2021).
Deep neural networks have reported great progress for vulnerability detection, with the recent Line- Vul paper (Fu & Tantithamthavom 2022) reporting 0.91 F1 score on a commonly used real-world vulnerability dataset (Fan et al. (2020), and many deep learning-based tools outperforming static analysis (Li et al.) 2018] Ding et al. [2022] Cao et al. (022). Current state-of-the-arr models use graph neural networks (GNNs) with unsupervised word embeddings and large pretrained transform- ers, which can perform well on this task. However, these models are large and expensive to train, and we showed that they did not generalize well beyond unseen projects. Empirical studies have discovered that the models can focus on spurious features which are not relevant to the cause of the bug. such as variable names (Chakraborty et al. 2021).
Inspired by the work done by Xu et al. (201)) and by Cranmer et al. 2020), we designed novel graph learning framework and embedding technique that is guided by program analysis algorithms of vulnerability detection, namely Dataflow Analysis (DFA). DFA computes the data usage patterns and relations in the control flow graph (CFG) of program and reports a vulnerability based on its root cause, i.c., whether the values and data relations collected from the program indicate the occur- rence of the vulnerable conditions. We explored an analogy between DFA and the GNN message- passing algorithm, and designed an embedding technique that encodes dataflow information each node of the CFG. Graph learning on such embedding thus simulates the dataflow computation in DFA.
We propose DeepDFA, a Deep Learning Framework guided by DFA, shown in Figure I Given the source code of a potentially vulnerable program, convert it CFG and encode the nodes

---

Under review as a conference paper at ICLR 2023
(defirtionn d,)
45 21 tazgo
Mg: St2 aalloc L.0
aeget
(aet ober Heangg)
Py:4c (urqu
(C peed(t))
Geeph leatning
U OUTJU
Datafoow analydis
(shor
Bacioo idelissiion de)
Ate ralleci1o segal2
GooLel 11gei-11
abelcco arqet-12 02
operater
UPDATAIN a) GEN, UINE KILL))
Geupl learning
Datafow analysis
(defittion
analysis-guided Graph learning
Source code
CFG abstract dataflow embedding
Figure 1: Overview of our DeepDFA approach
using an abstract dataflow embedding designed. Our abstract dataflow embedding represents variable definitions using the properties which are most important for vulnerability detection, based on domain knowledge from program analysis, e.g., the data types, API calls, constants, and operators present in the definition. Then, we apply graph learning and its message passing mechanism on the CFG edges to propagate such information across the edges of the graph, similar what is done in dataflow analysis. We then use the learned graph representation to classify whether the function is vulnerable or not.
Our evaluation shows that DeepDFA is significantly faster than our baseline models, both in terms of training and inference time. It only took minutes train and 1.92 ms/example for inference on CPU. Yet, DeepDFA still achieved top one for recall, ranked second for F1 and generalized to unseen projects the best, compared with other baseline models. Importantly, we show that DeepDFA embedding can be used with other models to further improve their performance. We created the top state-of-the-art vulnerability detection model by combining LineVul and DeepDFA and achieved 96.40 F1 score, 98.69 precision and 94.22 recall.
Along the research like GIN et al. (2019) and Cranmer et al. 2020), DeepDFA also demonstrated that guided by domain algorithmss, deep learning achieve better results with minimal resources. DeepDFA did not use any token and text level features which hardly reveal the cause of vulnera- bilities; instead it used abstract dataflow embedding to simulate the dataflow propagation (a casual domain algorithm) via graph learning, and used the idea of bitvector in dataflow analysis to achieve the efficiency of embedding and learning.
In summary, we made the following contributions in this paper:
1. We designed an abstract dataflow embedding to efficiently learn the program semantics relevant to vulnerability detection;
2. We applied graph learning the control flow graph (CFG) of the program and abstract dataflow embedding to simulate reaching definition dataflow analysis;
3. We implemented DeepDFA and experimentally demonstrated that DeepDFA outperforms baselines in vulnerability detection for effectiveness, efficiency, and generalization unseen projects;
4. We provided a comprehensive understanding on the analogy of dataflow analysis and graph learning, which can understand why DeepDFA performs well and efficient; and
5. We showed that DeepDFA can be used to improve other models, and we delivered the best vulnerability detection model in the state-of-the-art by combining Line Vul and DeepDFA.
2 RELATED WORK
A vulnerability is flaw in software program that be exploited to negatively impact the sumer or system (eve 2022). In the literature, vulnerability detection is typically framed as a binary classification problem. Devign /Zou et al. 2019), Re (Chakraborty et al. 2021). IVDetect (Li
2

---

Under review as a conference paper at ICLR 2023
et al. 202 and Line VD (Hin al. 2022) used GNN on program graph representations such as AST. CFG, and PDG, and annotated the nodes with unsupervised or pretrained word embeddings. ReGVD (Nguyen al.] (022) used GNN on co-occurrence graphs. We also used graph neural network, but we designed our graph learning framework using dataflow analysis as guidance. We therefore are able to understand why approach is efficient and performant.
Transformer models such as CodeBERT (Feng et al. 2020 and Vun (Fu & Tantithamthavom 2022) used token-based program representation pretrained on a large body of NL-PL pairs, and then fine-tuned for vulnerability detection. Using CFGs, our graph learning only propagates the information along semantically important edges instead of trying to learning the relations of each pair of tokens. Thus, our approach is sigmificantly more efficient. Since we have used a semantic- based embedding, we show that we can improve the performance of token based models. The most recent work, ContraFlow (Cheng et al., 2022), learns embeddings of def-use paths (an output of dataflow analysis), then predicts vulnerability detection using a transformer model. Our work directly emulates dataflow analysis and does not require expensive pretraining phase.
There are also models that used sequence and CNN architectures. VulDeePecker (Li et al. 2018) used BiLSTM on slices considering data dependencies. SySeVR (Li et al.) (02 used BiGRU on slices and adds data dependencies. Draper (Russell et al. 2019) used CNN and RF on a grid representation of source code.
While we were developing our analogy of DFA and graph learning, we found that Cummins et al. 2021 have proposed similar idea, but used it for device mapping and algorithm classification. However, their PrograMI. model used an unsupervised embedding which does not explicitly capture the domain-specific information used by DFA, and they did not target a specific dataflow analysis. Our abstract dataflow embedding captures the domain-specific information used by DFA and we applied reaching definition analysis, which directly linked to vulnerability detection.
Other relevant work that explores dataflow analysis and deep learning include: (1) VenkataKeerthy al. (2020) used the output of dataflow analysis, reaching definitions and live variables, learn flow-aware embeddings; and (2) Bielik et al. (2017) and Jeon et al. (2019) learned static analysis formulas from a dataset based on fixed language. None of these works aims develop a model for vulnerability detection.
DEEPDFA: EFFICIENT GRAPH LEARNING VIA DOMAIN GUIDANCE
Inspired by Xu al. (219)) and Cranmer et al. (2020), we aim to design a graph learning framework that can simulate and thus benefit from the domain algorithms for vulnerability detection.
3.1 DATAFLOW ANALYSIS FOR VULNERABILITY DETECTION
Dataflow analysis (DFA) is method for computing data usage patterns in program. In addition to compiler optimization, dataflow analysis is an important method for vulnerability detection. One instance of dataflow analysis, called reaching definition analysis, reports at which program points, a particular variable definition can reach. A definition reaches a node when there is a path in the CFG that connects the definition and the node, and the variable is not redefined along the path. The reaching definition analysis can detect a null-pointer deference vulnerability based on its root cause, when it identifies that definition of an NULL pointer reaches dereference of the pointer. Similarly, it is a causal step to detect many other vulnerabilities such as buffer overflows, integer overflow, uninitialized variables, double-fret and use-after-free (Cesare 2013).
DFA uses two equations to propagate the dataflow information through the neighbouring nodes in the CFG, namely meet-over-path and transfer function (Aho & Aho 2007). Meet-over-path aggregates the dataflow sets from its neighbours. Transfer function updates the dataflow set using the information available in node v. In the reaching definition analysis, the dataflow set is a set of definitions that reach program point. A simple approach of performing DFA is the Kildall method (Kildall 1973) (the full algorithm is given in Appendix B 3. It iteratively propagates the dataflow information to the neighbours of in the CFG, step time. The algorithm terminates when the dataflow information of all nodes stops changing, denoted a fixpoint. At termination, all nodes will incorporate the dataflow information from all other relevant nodes. This information is
3

---

Under review as a conference paper at ICLR 2023
then compared to the vulnerability conditions to determine whether vulnerability has occurred in the program.
3.2 ANALOGY OF GRAPH LEARNING AND DATAFLOW ANALYSIS
Figure Za shows a snippet of CFG where DFA performed. In the CFG, each node is statement, and each edge indicates the order of execution between two statements. The two dataflow equations that define reaching definition dataflow analysis are shown in the figure:
meet-over-path: IN[e0] U UEppred(r) OUT(u]
transfer function: OUT(v) GEN, U(TN(e) KILL.)
where [N' and [UT are the sets of dataflow located at the beginning and end of statement. GEN, and KILL, represent the dataflow generated (new definitions) and killed (overwritten defi- nition) in node
Figure shows an analogous behavior of graph learning. Graph learning performs a fixed number of iterations of paassgg-passing. At each iteration, each node aggregates information from its neigh- bors, and then updates its state to integrate the information. Here, the AGGREGATE and UPDATE functions serve similar role to the two dataflow equations of meet-over- and transfer function:
aggregate: at ACGRECATE(15 E pred(v)))
* update: ht UPDATEE(N
where at denotes the aggregated information from the neighbors, hy denotes the state of node after iterations of pesssge-passsing (analogous to OUT(e)).
d
X
de X _...
X
dx =...
[|
OUT
(G 1/u pred(()))
sEgreats)
X
dg X
OUT GEN, U(IN KILL.)
M = UPDATE(
(a) Dataflow Analysis
(b) Graph Learning
Figure Analogy of Dataflow Analysis and Graph Learning
3.3 GRAPH LEARNING TO SIMULATE DATAFLOW ANALYSIS
Based on the analogous behaviors of DFA and GNN, developed DeepDFA. It conducts graph learning on the CFG of a program, guided by the traditional DFA algorithm. In addition, we designed node embedding that can represent the dataflow set each node.
Designing A Abstract Dataflow Embedding: In dataflow analysis, we use a bit vector to represent the dataflow set at each node. A bit vector consists of n bits of Os and Is. Its length is the size of the domain. A bit is set to if its corresponding element is present in the set. In reaching definition analysis, the domain consists of all the definitions the program, and the bits are set to "1" if the corresponding definitions reach the node. For example, in Figure [ the program contains three definitions nodes U1, V3, and 04 so the reaching definition analysis uses a bit vector [o o 0] to initialize each node before the analysis. The bit vector represents OUT[w] in the dataflow equation. is updated at each step of propagation, and when the analysis terminates, the bit vectors for each node represent all possible definitions which reach that node.
The bit vector representation efficiently encodes program semantic features relevant to vulnerability detection. It can be quickly obtained at the node via lightweight program analysis. However, in graph learning, we cannot directly use it as the node embedding for two reasons. First, the domain of definitions is specific program. Different programs contain different variable definitions; their
4

---

Under review as a conference paper at ICLR 2023
sperssr
2: NONE
D. NONE
UNKNOWN
UNRNOVWN
code
arscbac
argos
Irdex
Abstract docsfiw
0
9
0: NONE
UNKOCVVI
Int
0
Build dictionaries for the properties
Extract properties
Abstract dstaflow embedding
Figure Abstract dataflow embedding generation
bit vectors have different lengths and the elements are not comparable. In graph learning, we want to extract dataflow patterns from all the programs and need to have "global" embedding that can be compared across programs. Second, bit vectors are changed in a discrete fashion during dataflow analysis; however, during graph learning, the node embedding needs be changed in continuous increments.
To address the challenges, we map all the concrete definitions in the programs in the training dataset to abstract definitions using the common properties of the definitions. Thus, the definitions from different programs can be compared and generalized. Using domain knowledge, we believe that the following four properties are important to the cause of vulnerability and can be shared across programs, so we use them to represent definition:
API call: the call to library or system functions used in the definition, e.g. malloc, strlen
Data type: the data type of the variable being assigned, e.g. int, chark.
Constant: the constant values assigned in the definition, e.g. NULL, -1, "foo".
Operator: the operators used in the definition, e.g. T.
We analyze large corpus of programs, e.g., training set, and collect top k frequently used API calls, data types, constants and operators to construct the dictionaries. For a particular definition, we identified its API call, data type, constant and operator, found their indices in the dictionaries, and converted each index to a vector using one-hot encoding. The generated embedding is matrix rows and 4 columns (each column is a one-hot vector representing we name Abstract Dataflow Embedding. Note that k a hyper-parameter of DeepDFA that we tune. Using larger k. we can represent more definitions using our embedding precisely, and of course, with larger overhead. If the property's 's value was not in the top-k, we used an UNKNOWN token represent its value; if the property is not present in the definition, we used NONE token.
of the 4 properties), which
Figure 3 describes an abstract dataflow embedding for an example dz: str = ma oc (10 argc), which used an API call malloc, with the constant 10, operator and data type char*. Contrasted with the 3-bit bit vector that represents concrete definition in data flow analysis, the abstract embedding is larger (5x4 elements), but it is still a fixed size, so it will scale any size of dataset without impacting the model's efficiency. The embedding keys have consistent meanings among all programs in the dataset so the model can capture the dataflow patterns across programs.
Graph learning guided by Dataflow Analysis: Given the CFG with abstract dataflow embeddings, DeepDFA performs message-passinn paasing on the CFG edges, which parallels the dataflow propagation in the DFA algorithm.
In Table I we stepped through the reaching definition analysis for the CFG example Figure in order to further demonstrate the analogy between the two, and to help understand how our graph learning framework can benefit from the DFA guidance: (1) we only focus on the semantic information relevant to the vulnerability (2) the rules simple and thus possible to learn (3) the iterative propagation can integrate information along the paths of a CFG to detect bugs based on its causes.
5

---

Under review as a conference paper at ICLR 2023
The row Iteration 0 shows the initializa- tion of each node in the reaching definition analysis. At iteration 1, the DFA updates ( OUT(va] and OUT((a) using the transfer function indicate that the new def- initions are introduced at the nodes. At it- eration 2, OUT(v) [iicluuing di) propagates to U2 and OUT jes (incluuing to va. through the CFG edges. At iteration
Table 1: OUT(u) at each iteration of DFA
Iteration
V1
2/2
F3
1/4
0
[0 a o
[0 0 0]
[0 0
0 0J
1
[1 1
[0 [
[0 1
[0 0 1]
2
o
0
[0
3
[1 0
0
[0 0]
[ 1
(including d2) propagates
3. the meet-over-path is used to combine OUT [2] and [UT[ Specifically, IN(ca]
U(OUT(v2),
OUT (UT(va]] computed as [1 0 V [o 0] [1 o]; then the transfer function combines
[1a] with GEN,,, v.. resulting in OUT(ps] = [1 [].
In graph learning, the analogous functions of AGGREGATE and UPDATE are learned using the neural networks. For example, in GGNN (Li et al. 2016), AGGREGATE is an MLP and UPDATE multiple neighboring nodes into one
GRU. AGGREGATE combines the representations vector, similar to meet-over-pathe UPDATE uses GRU to preserve forget portions of the previous state n3- and integrate new information from a',, similar to the operations performed in the transfer function. Within iterations, va will integrate the information from nodes V2 and vy (i.e., the definitions of di and d2 respectively) with its initial dataflow information dg-
After the DFA algorithm terminates, the final state of the nodes is used to detect vulnerabilities. The state of 24 is [1 1 1], which indicates that both dj and d2 may reach depending on the program values. Because the definition di str NULL can reach the dereference U4, we can conclude that this program has a null-pointer dereference vulnerability. Similarly, in graph learning, after a fixed number of iterations, all the node representations are combined using graph readout operation to produce graph-level classification for vulnerability detection.
As Cummins et al. (202T) noted, one minor difference between the two algorithms is that DFA iterates until it reaches a fixpoint, while GNN performs fixed number of iterations. Kam & Ullman (177) showed that DFA converges within d(G') + passes, where d the loop connectedness of G (defined in Cummins et al. (2021)). We set the hyperparameter t based on the best validation performance through our experimentation.
EVALUATION
We studied the 3 research questions: (1) Is DeepDFA effective for finding vulnerabilities? (2) How efficient is DeepDFA compared to other models? (3) Can DeepDFA generalize to unseen projects?
4.1 EXPERIMENT SETUP
To explore whether DeepDFA can advance the state-of-the-art, we cre- ated two settings, DeepDFA and DeepDFA+LineVull DeepDFA is implemented using the GGNN architecture (Li et al. 2016). DeepDFA+LineVul is created by concatenating Vine's RoBERTa encoder output to DeepDFA's graph stage output after readout. To avoid data leakage, we trained the abstract dataflow embedding on the training set. We used threshold of & = 1000, and it has covered (79.38% of the definitions in the test dataset. Future work may train the abstract dataflow embedding on large unsupervised dataset of code in order to learn a more general representation. See Appendix for the further details of our hyperparameters and training.
Model Implementation:
Dataset: In the recent literature (Fu & Tantithamthavorn 2022 Li et al. 2021a Chakraborty et al., 2021), vulnerability detection models are typically evaluated with Devign (Zhou et al.) 2019) or Big- Vul (Fan al.) 2020) datasets, both of which contains real- world open-source C/C++ projects. In our evaluation, we used the Big-Vur dataset because (I) it is bigger compared to Devign and consists of 188,636 functions, and 10,900 (6%) vulnerable labels, and (2) it reflects the imbalanced distribution of real-world code (Devign is a balanced dataset), with the minority of code containing vulnerabilities (Chakraborty et al. 2021).
We used the train/validation/eest splits of 80/10/10% generated by the Line paper (Fu & Tan- tithamthavorm, 2022). To address class imbalance, we undersampled the majority class (non-

---

Under review as a conference paper at ICLR 2023
vulnerable) of the training set without changing the validation and test sets. We used Joemn Ya- maguchi et al. 2014 to parse the code into its CFG representation. Because Joern cannot parse some programs (7%) the dataset, we used the rest of the 93% of the data in our experiments.
We also created a datasest to evaluate the generalization of the models. This dataset consists of the mixed-project and cross-project two settings. Our approach is to hold out 10k examples from randomly selected projects in Big- Vul as a validation and test set of the cross-project setting. We then randomly partitioned the remaining data into train/validdtionntess sets with 10k examples in the validation and test data, and the remaining examples in training. The second 10k examples can come from the same projects as the examples used in training, which we call the mixed project-reccttte setting. To mitigate the potential bias caused by the selection of projects, we repeated this process 5 times with different selections of the cross-project data and performed 5-fold cross validation.
Baselines: We compared against 7 non-transformer models: VulDeePecker, SySeVR, Draper, De- vign, Re ReGVD, IVDetect, and 2 transformer models: CodeBERT and Line Vu These models are developed recently with different architectures and represent the state-of-the-art of vul- nerability detection models. See Section 2 for an overview of the models and Appendix A for the details of our reproductions.
Metries: We evaluated model performance and efficiency using metrics from the literature. Specifi- TP
cally, Precision reports the portion of positive predictions which were correct: P TP+FP. Recall
TP calculates the portion of positive examples which were recalled: R = TP4FN- F1 is the harmonic mean between Precision and Recall: F1 2 P-R We used F1 to decide the highest performing
P=R
model because it balances precision and recall, which are both important in an imbalanced dataset.
For efficiency, we reported the Training time: the wall-clock time to execute one training run with one validation run per epoch, and the Inference time: the average wall-clock time to predict for one example. We also used MACs: the average number of
operation (Tan & Le| 2020; Howard et al. 2017) to predict for one example; this measures the performance independently of the computing platform. Finally, we listed the model size in terms of Parameter count: the number of trainable parameters in the neural network model.
We trained the models times with different random seeds and reported the mean score and standard deviation for each metric. We ran the runtime benchmarks an AMD Ryzen 1600 3.2 GHz processor with 48GB of RAM and an Nvidia 3090 GPU with 24GB of GPU memory.
4.2 RESULTS: EPFECTIVENESS
Comparison with non-transformer models: As shown in Table 2a DeepDFA performed sig- nificantly better than the baselines on Fl score; the average difference was 47.51. DecpDFA also achieved a significantly higher Recall score compared to other models, and its Precision score ranked 4th out of the 7 baselines. Compared to the other 6 models, DeepDFA reported significantly lower variances for all the three metrics.
DeepDFA's abstract dataflow embedding encodes the semantic features (dataflow sets) of programs relevant to the vulnerability detection. On the other hand, the other models used unsupervised word embeddings such as Word2Vec, Doc2Vec, and GloVe as node embedding, which do not represent a causal relationship with vulnerabilities. Among the bascline tools, Devign and Veal used CPG which combines AST, CFG, and PDG, while DecpDFA used only the CFG, AST and PDG information are useful to detect certain classes of bugs (Yamaguchi et al. 2014), but compared to DeepDFA which directly simulate the dataflow analysis the CFG, these approaches less accurate to model data relations needed for causal vulnerability detection.
Comparison with transformer models: CodeBERT and Vun are the best among all the trans- former models that detect vulnerabilities in the Bug dataset (see Appendix C for the detailed numbers). Table 2h shows that DeepDFA reported the highest recall and the smallest variances compared to CodeBERT and Vul. It ranked second for Fl and performed significantly better than CodeBERT in Fl. We achieved the state-of-the-art performance of Fl score of 96.40 (3.17
https: /ttt:t//jjeeeeeee
We could not reproduce LineVD and ContraFlow on Big- for function-level vulncrability detection.
We used DeepSpeed profiler measure MACs. https: /www. deepspeed i
7

---

Under review as a conference paper at ICLR 2023
Table DeepDFA outperformed the baselines and can be used further improve the existing model performance. Note that VulDeePecker, SySeVR, Draper, and IVDetect performance were directly taken from the IVDetect paper (Li al. 2 so we not report the variance.
(a) Comparison with non-transformer models.
(b) Comparison with transformer models.
Model
F1
Model
F1
R
VulDeePecker
12.00
49.00
19.00
CodeBERT
21.04
68.48
12.91
SySeVR
15.00
74.00
27.00
(6.72)
(11.76)
(5.51)
Draper
16.00
48.00
24.00
DeepDFA
68.26
53.98
92.81
IVDetect
23.00
72.00
35.00
(0.16)
(0.06)
(0.40)
Devign
26.85
29.00
25.03
Lun
93.23
97.32
89,48
(0.97)
(0.38)
(1.67)
(0.31)
(0.66)
(0.42)
ReVeal
32.94
34.27
31.73
DeepDFA+Line Vul
96.40
98.69
94.22
(0.75)
(1.58)
(0.65)
(0.13)
(0.28)
(0.46)
ReGVD
19.15
63.67
11.33
(2.65)
(4.43)
(1.94)
DeepDFA
68.26
53.98
92.81
(0.16)
(0.06)
(0.40)
improvement), Precision score of 98.69 (1.37 improvement), and Recall score 94.22 (4.75 im- provement), by adding DeepDFA's embedding to Vun. Since DeepDFA is lightweight, it can be added to improve Vul little cost. The improvement suggests that DeepDFA's embedding can provide additional useful information even for very large pretrained models trained from the tokens of programs.
DeepDFA does not use any text-level/token information such variable and function names, and yet has achieved very good performance. We believe that leveraging the domain algorithm of reaching definition analysis to guide graph learning indeed plays an important role, and the em- bedding indeed encodes semantic features (c.g., data relations) that are important for vulnerability detection. Interestingly, such features also improve the top-performing model along all metrics. We further believe that the examples which DeepDFA predicted incorrectly could be due to the fact that reaching definition analysis cannot handle all types of vulnerabilities. Thus, by adding other dataflow analyses such as live variable analysis, DeepDFA can further improve its performance. We will leave this step our future work.
4.3 RESULTS: EFFICIENCY
Table 3: Runtime of the different models.
Train time (ms)
GPU Inference
CPU Inference (ms/example)
Model
MACs/example
(ms/example)
Line Vul
10h19m
11.06
1068.2
48.32 B
DecpDFA
9m
4.64
5.8
40.27 M
D
10h40m
15.36
1571.5
48.32 B
In Table
we present the runtime comparison of DeepDFA and Vune Here, we did list other models because their performances are much worse (shown in Section 4.2), and they took hours train, compared to DeepDFA which finished training in 9 minutes (excluding data preprocessing time). In Table 4
we also listed the sizes of the models in terms of the number of parameters.
Compared to LineVul, DeepDFA took 69x less time to train, had 2.4x faster inference on GPU and 185x faster inference on CPU, and used 333x fewer parameters. DeepDFA had the least number of parameters all models, equal to 67% of the smallest model (ReVeal) and 0.3% of the largest baseline model (Line These results consistently indicate that DeepDFA excels in its efficiency compared to other models. This possible because DeepDFA is based on the dataflow analysis's compact bitttettionbbivveooooro.. which represents semantic information in bits and thus is more efficient compared to tokenized strings. DeepDFA propagated information along only the domain- specific CFG edges, rather than associating every pair of tokens an exhaustive fashion.
8

---

Under review as a conference paper at ICLR 2023
DeepDFA's short inference time due to low number of MAC operations enables its use in non-GPU environments (which are common for software development) where larger models such VineVul may not be easy to be deploy. DeepDFA's short training time enables techniques like per-project tine-tuning and hyperpa- rameter tuning, which would be much more costly with Vul's training time of over 10 hours. Because of DeepDFA's ' small parameter count, it is ideal for resource-limien computing plat- forms such mobile devices, where large models cannot be used (Howard et al. 2017).
Table 4: Size of the models.
Model
parameters
LineVul
125,238,531
DeepDFA
375,938
DDFA+LV
125,679,236
IVDetect
924.165
Devign
1,148,553
Re
560,291
ReGVD
124,794,500
CodeBERT
124,646,401
4.4 RESULTS: GENERALIZZTION
We compared the cross-project (shown as Cross F1) and the mixed-project (shown Mixed FI) test F1 scores to evaluate the models' capabil- ities of generalizing over unseen projects. Ta- ble 5 presents the highest-performing baseline model VineVul compared to DeepDFA (the re- sults of other models are available in Appendix D) Shown under Column A FI, DeepDFA reported the most consistent performance be- tween cross-project and mixed-project settings. When applying to unseen projects, DeepDFA
Table 5: How do the models handle unseen projects? Note the performance drop (A F1) from the cross-project mixed-project setting.
Model
Mixed F1
Cross F1
A Fl
Line Vul
84.03
71.37
- -12.66
DeepDFA
70.49
68.58
-191
DDFA+LV
87.89
71,88
-16.02
only dropped 1.91 (2.7%) Fl score, compared to 12.66 (15.1 %) drop for Line and 16.02 (18.2%) drop for DeepDFA+LineVin Vul. DeepDFA+LineVuu Vul reported the best performance for both the mixed- project and cross-project datasets, improving on LineVull mean Fl score by 3.74 and 0.54 points respectively. Although DeepDFA did not rank the best in terms of the absolute values, its gener- alization was significantly better than the other models because its performance dropped the least from the mixed-project set to the pross-project set.
We believe that DeepDFA generalizes better because it does not rely on superfluous features that may exist token and text level such as variable names and function names as reported by the previous research (Chakraborty et al.] [202T). For example, Line Vul takes input tokens, which often differ between different projects and thus are hard to generalize their patterns. On the other hand, our abstract dataflow embedding encodes the usage patterns of commonly used API calls, operators, constants, and data types. Such patterns can be directly related to the cause of the vulnerabilities, and thus might help DeepDFA generalize better unseen projects.
5 CONCLUSIONS AND FUTURE WORK
We propose DeepDFA, an efficient graph learning framework and embedding technique guided by dataflow analysis for vulnerability detection. Our abstract dataflow embedding incorporates domain knowledge and extract semantic features of data usage patterns of commonly used API calls, opera- tions, constants, and data types that potentially capture the causes of the vulnerabilities. The graph learning emulates the Kildall method of dataflow analysis using the analogous message-passing algorithm. Our experimental results confirm that DecpDFA is performant and efficient, and gener- alizes better than other models. DeepDFA can be trained in minutes and yield a smallest model compared to all other baseline models, yet still achieving the top ranking for recall and no. 2 ranking for FI. We show that DeepDFA can be used to improve other models. By combining LineVul and DeepDFA, we delivered the best vulnerability detection model in the state-of-the-at at 96.40 Fl score, 98.69 precision and 94.22 recall. In the future, we plan to incorporate the guidance of other dataflow analyses such as live variable analysis for graph learning. These dataflow analyses also used for vulnerability detection(Cesann (013).. We also plan to explore the application of DeepDFA to precisely pinpoint the vulnerability location at specific lines in the code.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All code and data used in this paper are available at hhttps://iigshate com/ /// e 995b4434500390d17 along with trained model weights and scripts to run the experiments as we ran them.
Section 4.1 documents our benchmarking environment.
Appendix documents the hyperrpaaameters used for the GGNN model and abstract dataflow embedding.
We used Joern version 1.1.1072 to parse the CFG, available at htttts:://oornn.....
REFERENCES
Browse vulnerabiliities date.
wwwp:.rreeeraal.....oooroossss-e 2021.
2021. Accessed November
IBM Cost of Data Breach Report 2021, 2021. Accessed October 29 2021.
List of data breaches.
en httppp//w..gg......///....a.aaa. 29 2021.
Oct 2021, Accessed October
"vulnerability", 2022. URL http://sss/w....cee/////eeareaaa
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Chang. Unified Pre-training for Program Understanding and Generation, April 2021. URL 21 2103.90 06333 arXiv:2103.06333 [es].
Alfred V. Aho and Alfred V. Aho (eds.). Compilers: principles, techniques, & tools. Pear- son/Addison Wesley, Boston, 2nd edition, 2007. ISBN 978-0-321-48681-3. OCLCC ocm70775643.
Pavol Bielik, Veselin Raychev, and Martin Vechev. Learning a static analyzer from data. Lec-
ture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and
Lecture Notes in Bioinformatics). 10426 LNCS(ii))23332553, 2017. ISSN 9783319633862. doi:
00.1007998-33311--388---
Sicong Cao, Xiaobing Sun, Lili Bo, Rongxin Wu, Bin Li, and Chuanqi Tao. MVD: Memory-Related Vulnerability Detection Based on Flow-Sensitive Graph Neural Networks. arXiv:2203.02666 [cs], March 2022. doi: 10.1
URL URL.hcpp///axxi.oor//bb.:
02660 arXiv: 2203.02660.
Silvio Cesare. Bugalyze.com Detecting Bugs Using Decompilation and Data Flow Analysis. In BlackHat USA, pp. 9, 2013.
Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep Learning based Vulnerability Detection: Are We There Yet. IEEE Transactions Software Engineering, pp. 1- 1, 2021. doi: 10. 20109/9s.222113087744
Xiao Cheng, Guanqin Zhang, Haoyu Wang, and Yulei Sui. Path-sensitive code embedding via con- trastive learning for software vulnerability detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, pp. 519-531, New York, NY, USA, July 2022. Association for Computing Machinery. ISBN 978 978-1-450339939999 doi: 10. 00.1145/3333667355444 333333337777333377. URL https:
100 11..0..11533300
35334771
Miles Cranmer, Alvaro and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, ( 2020.
Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel,
10

---

Under review as a conference paper at ICLR 2023
Chris Cummins, Zacharias V Fisches, Tal Ben-Nun, Torsten Hoefler, Michael F P O' O'Boyle, and Hugh Leather. ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. Proceedings of the 38th International Conference Machine Learning, 139(M)):244--255, 2021. URL
hgs.mlr.................. press/v19/ pre
cummi ns21a html
Yangruibo Ding, Sahil Suneja, Yunhui Zheng, Jim Laredo, Alessandro Morari, Gail Kaiser, and Baishakhi Ray. VELVET: a noVel Ensemble Learning approach to automatically locate VulnEr- able saatements. arXiv:21 /2. 10899 [cs], January 2022. URL http:///rxiv..ort
2112. 10893
arXiv: 2112.10893.
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen, A CIC++ Code Vulnerability Dataset with Code Changes and CVE Summaries. Proceedings 2020 JEEE/ACM 17th International Confer- Mining Software Repositories, MSR 2020, pp. 508-512, 2020. ISSN 9781450379571.
doi: 10.1 00.114//33795973388555
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A Pre-Traine Model for Programming and Natural Languages. pp. 1547, 2020.
doi: 1..18655/1/2222 2ttp:///.222....nnnnnnoooo findings-emnlp em..p.139. 139.
Michael Fu and Chakkrit Tantithamthavorn. LineVul: A Transformer-basee Line-Level Vulnerabil- ity Prediction. 2022 IEEE/ACM 19th International Conference on Mining Software Reposito- ries (MSR), pp. 608-620, May 2022. doi: 10111457524442..528555..
ISSN: 2574-3864.
Hazim Hanif and Sergio Maffeis. VulBERTa: Simplified Source Code Pre- Training for Vulnerability Detection. Technical Report arXiv:2205. 12424, arXiv, May 2022. URL http::tttt:/ttt org/
abs/ 2a0s/2205.c0 1055.12444 12424 arXiv:2205. 12424 [cs] type: article.
David Hin, Andrey Kan, Huaming Chen, and M. Ali Babar. LineVD: Statement-level Vulnerability Detection using Graph Neural Networks. arXiv:2203.0511 [cs], March 2022. URL http:
//axxiv.org/abe..
05181
arXiv: 2203.05181.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, April 2017. URL http:://ars.v..rr/ab///// 04861 trXiv:1704.04861 [cs].
Minseok Jeon, Sehun Jeong, Sungdeok Cha, and Hakjoo Oh. A Machine-Learning Algorithm with Disjunctive Model for Data-Driven Program Analysis. ACM Transactions on Programming Lan- guages and Systems, 41(2):13:1 June 2019. ISSN 0164-0925. doi: 10.1145/3293607.
/ectts::/doi.... URL httpt::t////. /oii wrr.i.oorg/lin 11t.:////...1111299111
John B. Kam and Jeffrey D. Ullman. Monotone data flow analysis frameworks. Acta Informatica,
(13):305-177, September 1977. ISSN 1432-0525. doi: 10.10077BBF02903339, URL https:
//dei.org//// 1007 00290033
Gary A. Kildall. A unified approach to global program optimization. In Proceedings of the Ist
annual ACM SIGACT-SIGPLAN symposium on Principles of programming languages POPL
73. pp. 194-206, Boston, Massachusetts, 1973. ACM Press. doi: 10.1145/51222.5522955 URL http::/ acm www.mmoog///caaiiioooooooooo wttp::ecccc.ooiddodddoe10
512945
Yi Li, Shaohua Wang, and Tien N. Nguyen. Vulnerability detection with fine-grained interpretations. pp. 292-303, 2021a. ISSN 978 188443885626, doi: 10. 00.1145/466224444668777
Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. 4th International Conference Learning Representations, ICLR 2016 Conference Track Proceedings, (1):1-20, 2016.
Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. VulDeePecker: A Deep Learning-Bases System for Vulnerability Detection. (February), 2018. doi: 14.14/22/ssss 2012 2018.23155.
11

---

Under review as a conference paper at ICLR 2023
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. SySeVR: A Frame- work for Using Deep Learning to Detect Software Vulnerabilities. JEEE Transactions on De- pendable and Secure Compuring, pp. 1-15, 2021b. doi: 10.1099TDDSC22221335512222
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang. Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A Machine Learming Benchmark Dataset for Code Understand- ing and Generation, 2021. URL htt.:ttp:///axxi..or//////a//a www./r////////////////..//eeeees 2102. 04664
Van-Anh Nguyen, Dai Quoe Nguyen, Van Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. ReGVD: Revisiting Graph Neural Networks for Vulnerability Detection. March 2022. URL
https: htttps://ppeneeviee net/ Eorumm?i--BUSeniiiWkt.
Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Mare McConley. Automated Vulnerability Detection in Source Code Using Deep Representation Learning. Proceedings 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, pp. 757-762, 2019. ISSN 9781538668047. doi: 10.
1109/CCMLA.208..00
Publisher: IEFF.
Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking Model Scaling for Convolu- tional Neural Networks, September 2020. URL hutt:://arxiy>
arXiv:1905. 1946 [es, stat] version: 5.
S. Venkhta Rohit Aggarwal, Shalini Jain, Maunendra Sankar Desarkar, Ramakrishna Upadrasta, and Y. N. Srikant. IR2Vec: LLVM IR based Scalable Program Embeddings. ACM Transactions on Architecture and Code Optimization, 17(4):1-27, December 2020. ISSN 1544- 3566, 1544-3973. doi: 10.1145/3418463, URL httttt:///// arwiv org http//aas11990 06228 arXiv:1909.06226 [es].
Keyulu Xu, Stefanie Jegelka, Weihua Hu, and Jure Leskovec. How powerful are graph neural networks? 7th International Conference on Learning Representations, ICLR 2019, pp. 1-17, 2019.
Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vulner- abilities with code property graphs. Proceedingy IEEE Symposium Security and Privacy, pp- 590-604, 2014. ISSN 9781 479946860. doi: 10. 109/SP..01444
Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. D2A: A Dataset Built for Al-Based Vulnerability Detection Methods Using Differential Analysis. pp. 111-120, 2021. 00020.
doi: 10.1 (0.1100/ssseeseii2556000000
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks. Advances in Neural Information Processing Systems, 32:1-11, 2019.
12

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS
Hyperparameter
Value
(learning rate)
le 3
L2 weight
le
(threshold)
1000
t (number of GNN steps)
Hidden size
32
# output layers
Batch size
256
Table 6: Hyperparameters used for training in RQ1.
We trained DeepDFA for 25 epochs and evaluated the checkpoint with the lowest validation loss. We listed our hyperparameters in Table
We trained DeepDFA+LineVul with the same method LineVul; trained for 10 epochs and evaluated the checkpoint with the highest validation Fl score.
For the baselines,
We could not reproduce VulDeePecker. SySeVR, Draper, IVDetect, repeated the performances reported in Li al. (2021a). Their mcasurements may vary slightly from our reproduction.
We confirmed with Chakraborty et al. (2021) that our results fixed data leakage bug in the original implementation, so the results we report may differ from the original paper.
Zhou et al. (2019) did not release their model code, we reproduced Devign from the third-party implementation released by Chakraborty et al. (202T) (https: /ottps://github
com/ sa kat 077 DDvigg).
Hin et al. [2022) did not report function-level metrics for LineVD, and could not repro- duce their statement-level performance from their model code, so we did not compare with their approach.
Cheng al. 2022) did not release their model code and evaluated different dataset than ours, so we did not compare with their approach.
13

---

Under review as a conference paper at ICLR 2023
B THE DATAFLOW ANALYSIS ALGORITHM
Algorithm flstss the generic Kildall method for dataflow analysis. The meet-over-path 7 and trans- fer function must be specified in order compute a specific dataflow analysis.
Algorithm The round-robin iterative algorithm for dataflow analysis (i.e. Kildall method)
Given: graph G E), meet-over-path r. transfer function initial states eo for v in do
OUT < [v]
initial state for each node
end for
changed +- True while changed do
denotes whether OUT[u] changed for any
changed+ <- False
for in V do
IN[a] f n (epred(e)
OUT
D Do AGGREGATE/T & Do UPDATE/S
OUT [u] + ((N
if YUT K OUT then
OUT + OUT [v]
changed True
> If any OUT(e) changed, iterate again; otherwise terminate
end if end for end while
14

---

Under review as a conference paper at ICLR 2023
C
EFFECTIVENESS RESULTS ON THE FULL BIG- VUL DATASET
Of the transformer models, we evaluated CodeBERT and Line Vul our experiment for Section 4.2
Table 7: Initial trial run of performance on 100% of the Big- Vul dataset.
Model type
Model
F1
Precision
Recall
29.33
32.83
26.59
GNN
Devign
(6.58)
(5.55)
(7.25)
33.6
33.08
34.67
GNN
ReVeal
(0.69)
(3.49)
(3.8)
24.49
63.76
15.26
GNN
ReGVD
(3.16)
(3.54)
(2.71)
22.68
67.79
19.11
Transformer
CodeBERT
(8.12)
(4.9)
(3.39)
91.58
95.99
87.55
Transformer
LineVVl
(0.49)
(0.85)
(0.49)
VulBERTaMLPP
1.75
19.33
0.92
Transformer
Hanif & Maffeis
2022)
(3.03)
(33.49)
(1.59)
VuIBERIaCNN
10.59
5.59
100
Transformer
Hanif & Maffeis
2022)
(0)
(0)
(0)
PLBART
25.35
61.84
16.18
Transformer
(Ahmad et al.
2021
(3.74)
(6.54)
(3.52)
D CROSS-PROJECT EVALUATION RESULTS ON THE FULL BIG- VUL DATASET
We only evaluated Line Vul our experiment for Section 4.4 because its absolute performance was significantly better than the other models on our initial trial with 100% of the Big- Vul dataset (See Table 8
Table 8: Initial trial run cross-project evaluation with 100% of the dataset.
Model
Mixed-projee project F1
Cross-project F1
A F1
Line Vul
84.07
71.81
-12.6
Devign
24.32
14.26
-10.06
ReVeal
28.24
6.83
-21.41
ReGVD
33.61
21.70
-1199
CodeBERT
24.14
4.98
-19.16
15


Reviews:
[{"review_id": "uE1twn3DSxJ", "reviewer": "Reviewer_FYuz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a GNN based approach for vulnerability detection. The control flow graph (CFG) of a program is augmented with dataflow information and graph learning is applied to predict vulnerability of the program. This work proposes an “abstract dataflow embedding” to represent important dataflow properties that are relevant to vulnerability detection and uses message-passing of the GNN model to mimic the dataflow analysis algorithm.  Evaluation of the proposed model on publicly available dataset shows improved performance compared to several state-of-the-art approaches.", "summary_of_the_review": "This work inspires the use of dataflow information in GNN based vulnerability detection. However, the proposed approach has reduced novelty. Also, the evaluation and description of the overall system lacks clarity.", "strength_and_weaknesses": "Strengh\n\nDataflow analysis has been used as a useful tool for vulnerability analysis. Augmenting dataflow analysis to improve performance of ML based vulnerability detection approaches seems reasonable.\n\nWeakness\n\n- The use of GNN to model CFG of programs for vulnerability detection has already been explored by previous works (as mentioned in the related work) . So, the novelty of this work is restricted to ‘encode dataflow analysis in CFG’ for graph learning. However, it is not clear how (or if) the proposed embedding is more effective than the already proposed embedding approach in PrograML. There is no comparative discussion/evaluation that demonstrates the superiority of the proposed approach.\n- It is not clear if the proposed approach detects vulnerabilities at the program/function/line level. Baseline methods seem to include detectors that focus on both function and line level.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dAOeP_YKHV9", "reviewer": "Reviewer_x4r7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors present DeepDFA, a dataflow analysis-guided graph learning framework and embedding that use program semantic features for vulnerability detection. In experiment, they show DeepDFA ranked first in recall, first in generalizing over unseen projects, and second in F1 among all the state-of-the-art models they experimented with.", "summary_of_the_review": "The authors explored to use graph learning for vulnerability detection, but the novelty is minor:\n(1)The analogy of dataflow analysis and graph learning is somewhat trivial especially when there are existing works, for example“ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. ICML 2021” as this paper cited.\n(2)Moreover, the dataflow embedding in this paper, which only select API calls, data type, Constant, and Operator, seems not strong enough to represent dataflow. \n(3)The author shows experiment details, but can be improved\n\n", "strength_and_weaknesses": "Strength:\nThe paper is easy to follow in general.\nVulnerability detection using graph learning is an important problem.\n\nWeaknesses:\nThe authors explored to use graph learning for vulnerability detection, but the novelty is minor:\n(1)The analogy of dataflow analysis and graph learning is somewhat trivial especially when there are already several existing works, for example, “ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. ICML 2021” as this paper cited.\n\n(2)Moreover, the dataflow embedding in this paper, which only select API calls, data type, Constant, and Operator, seems not strong enough to represent dataflow. There is also other more important information, for example, tainted value or expressions, path constraint, and so on. Take the example in Figure 1, as the authors explained at the end of section 3.3, they show a null-pointer dereference vulnerability because the definition d1:str=NULL can reach the dereference at v4. How about when there is an if statement that checks whether str == NULL, if it does, program will terminate. Will DeepDFA also report a null-pointer dereference vulnerability since path constraints are omitted in the dataflow embedding? \n\n(3)The evaluation also should be improved, both in setting and description. \n\n(3.1) I wonder what level and what kinds of vulnerability DeepDFA is able to detect? \nIn evaluation, the authors did not compare with ProGraML, which is the most related to this work. Although ProGraML was evaluated on device mapping and algorithm classification, ProGraML is also able to do data flow tasks (referring to ProGraML’s evaluation), such as Reachability analysis, Dominance analysis, liveness analysis, and so on. Since the two works are so related, it would be better to compare them in Reachability analysis, Dominance analysis, liveness analysis, and so on. If possible, also compare the ability of vulnerability detection by self-implementation based on ProGraML.\n \n \n(3.2) As I know, CodeBERT is a model for natural language code search and code documentation generation, how the authors apply CodeBERT for vulnerability detection, and what kinds of vulnerability can be detected? It would be better to give more details.\n \n \n(3.3) In the evaluation, the authors only show F1 score, precision, recall. What kinds of vulnerabilities are detected in the experiments? It would be better to show more details of the vulnerabilities.\n \n \n(3.4) I noticed there are some closely related works not included and discussed:\n“Learning to Represent Programs with Graphs, ICLR2018”\n\"Learning semantic program embeddings with graph interval neural network. OOPSLA 2020”\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "_UB01NPQQDk", "reviewer": "Reviewer_EyDV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DeepDFA, an efficient graph neural network architecture based on the dataflow information of the program for vulnerable function detection. It develops the abstract dataflow embedding to represent the program. The key insight is that such representation enables the graph learning algorithm to behave similarly to the dataflow analysis process, a classical static analysis approach to detect vulnerabilities. Experimental results demonstrate that DeepDFA is fast in training and inference and achieves the best results when combined with an existing baseline.", "summary_of_the_review": "I like the idea of this paper that incorporates the dataflow analysis as the inductive bias of neural networks for program analysis. However, I think the paper needs to address the following concerns.\n\nIt is unclear why the properties used to represent each node in the graph mismatch the information used in actual dataflow analysis. As described in Section 3.3, the authors used 4 properties of a node without including the variables in the program, but they are critical to dataflow analysis. Without incorporating variables, it is hard to argue that the graph learning algorithm is still learning dataflow analysis.\n\nSimilar to variable names, which the authors discard as they are program-specific, the 4 properties used to represent statement nodes are also domain-specific. API calls, type strings, constants, and operators might change across different programming languages, and their distribution might vary significantly across projects. How do these properties generalize to unseen ones? Are there experiments to ablate their effectiveness?\n\nWhat is the overhead of extracting the extra information of the program, i.e., abstract dataflow embedding, over using raw code tokens? It is helpful to show the tradeoff of increased data processing effort for improved training/inference efficiency.\n\nDataflow analysis algorithm assumes termination when reaching a fixed point. The authors also mentioned that the number of iterations depends on the graph size (d(G)+3). Why did the authors decide to fix such hyperparameters based on a validation set?\n\nWhat are the vulnerability types in the evaluated dataset -- is dataflow analysis indeed enough to analyze all the vulnerability types? If yes, as a baseline, the authors should report the performance of the classic dataflow analysis and argue why we need an alternative based on machine learning. If not, it is important to show in which case dataflow analysis fails. How does the model perform in these cases? What makes the model outperform the standard dataflow analysis (if this is true), if it simply learns the dataflow analysis?\n\nMinor:\n\nThe scale of the metrics described and used in the paper should be consistent. For example, the authors use 0.91 F1 scores in the introduction but use 96.4 F1 scores in the other places.\n\nThe authors of the citation \"Compilers: principles, techniques, & tools\" is incorrect.\n\nSection 2, I do not think that \"propagates the information along semantically important edges\" are realized in this paper. This paper proposes to propagate information on control flow graphs, but only control flows do not fully capture all the dependencies, e.g., data dependencies. I am not fully convinced that such a setup is \"significantly more efficient\" than transformers.", "strength_and_weaknesses": "+ Design: analogies of the dataflow analysis algorithm and the message passing algorithm in graph neural networks are intuitive and novel.\n+ Results: the dataflow embedding improves the efficiency, precision, and generalizability of graph neural networks in vulnerability detection.\n\n- Unclear design choices of abstract dataflow embedding, in which the variable -- the key part in dataflow analysis - is not included.\n- Evaluation should show the vulnerability types and how dataflow analysis identifies them. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3gBkz4RA1s1", "reviewer": "Reviewer_ePbc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dataflow analysis-guided graph learning approach namely DeepDFA for vulnerability detection. It encodes the node from control flow graph using an abstract dataflow embedding. Based on the vectorized node embedding, it further applies GGNN on the CFG for message passing. Then a graph representation is used to classify whether a function is vulnerable or not. By the experimental results, it confirms that DeepDFA is significantly faster than the baseline models. Furthermore, it also confirms that the designed embedding approach is also general to other vulnerability detection models. ", "summary_of_the_review": "Although the idea to compare data flow analysis with GNNs is interesting, the technical novelty is not enough. Furthermore, the reason behind the embedding layer is also unclear. ", "strength_and_weaknesses": "Strength:\n\n- The related work is comprehensive, though there are some missing related works, it still cover current the most deep learning techniques for vulnerability detection. \n- The idea to compare data flow analysis to message passing in GNNs seems interesting. \n- This paper is easy to follow.\n\nWeaknesses:\n\n- This paper uses data flow for vulnerability dection. I agree that there are many vulnerability types caused by data dependencies, however how to deal with other vulnerabilites that are caused by other dependencies such as control dependencies?\n\n- Why choosing four properties such as API call, Data type, Constant and Operator? Are these properties enough for vulnerability detection? There appears to be insufficient evidence to justify this choice.\n\n- What is the novelty of model except the designed the abstract dataflow embedding? It seems that this paper just replace the embedding layer in GNNs to the abstract dataflow embedding for vulnerability detection. Furthermore, since the dataflow embedding size can be controlled by the hyper-parameter k, hence the model size and parameters will denifitely smaller than others. \n\n- Why choose GGNN for the model architecture? Can we use other GNN variants for the replacement.\n\n- I suggest authors to combine Table 2 (a) and (b) for better presentation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject