Exploring General Intelligence of Program Analysis for Multiple Tasks
Authors: Yixin Guo
Pengcheng Li
Yingwei Luo
Xiaolin Wang
Zhenlin Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: GNN
program analysis

Abstract: Artificial intelligence are gaining more attractions for program analysis and semantic understanding. Nowadays, the prevalent program embedding techniques usually target at one single task, for example detection of binary similarity, program classification, program comment auto-complement, etc, due to the ever-growing program complexities and scale. To this end, we explore a generic program embedding approach that aim at solving multiple program analysis tasks. We design models to extract features of a program, represent the program as an embedding, and use this embedding to solve various analysis tasks. Since different tasks require not only access to the features of the source code, but also are highly relevant to its compilation process, traditional source code or AST-based embedding approaches are no longer applicable. Therefore, we propose a new program embedding approach that constructs a program representation based on the assembly code and simultaneously exploits the rich graph structure information present in the program. We tested our model on two tasks, program classification and binary similarity detection, and obtained accuracy of 
80.35% and 45.16%, respectively.

Under review as a conference paper at ICLR 2022
EXPLORING GENERAL INTELLIGENCE OF PROGRAM ANALYSIS FOR MULTIPLE TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
As the program scale ever grows, program analysis tasks become increasingly diverse and complex. Effective barming-based program analysis tasks need to consider many factors including also running platforms and compilation configurations. Existing works are of- tentimes on a basis of source code or abstract syntax tree, but unfortunately lack sufficient program structure and semantic information to unveil program insights. To this end, we propose a new program analysis model that takes advantage of great power of graph neural networks from the level of assembly code, which can solve both traditional program semantics- basee problems and compilation-relates problems, by fusing the semantics of the program's 's instructions and existence of multiple graph structures in programs. We tested it on two program-level tasks and achieved as high accuracy as 82.58% and 83.25% respectively.
only program structures and semantics but
INTRODUCTION
With the development of information technology, computer programs are used in an increasingly wide range of fields. As the number and variety of programs continue to expand, the task of ana- lyzing programs becomes more significant and complex. Common program analysis tasks include program classification, duplicate code detection, making programming suggestions (syntax and al- gorithms), program vulnerability analysis, bug detection, cross- language program translation, code annotation, etc. Program analysis is not simple problem and many factors bring complexity it. First, program functions are not implemented in unique way; apparently completely different codes may implement the same functions, and similar codes may implement quite different functions. Sec- ond, programs may be written in different languages and run various platforms. The same code compiled by different compilation configurations may also have outstanding differences in terms of correctness and performance. All of these the major factors to consider during program analysis.
In recent years, more and more researchers have introduced machine learning approaches to solve complex program analysis problems. Earlier approaches treat codes as shallow textual structures, e.g., sequences of tokens (Allamanis et al. 2016a, Hindle et al. 2016 Allamanis et al. 2016b) PPrddel & Sen| 2018), and analyze them with natural language processing models. These approaches ignore the rich and explicit structural information that exists in program. In addition, since many variables are defined and used not adjacent each other or even far apart, this "def-use" relations are difficult to capture by this kind of model. However, such long-distance dependencies, can be reflected the structural information of the program.
By abstracting programs as graphs and using graph neural networks (GNN) for program analysis tasks, better use can be made of these structure information that are readily available in programs. Graphs such as control flow graph, call graph, and data flow graph reflect the control flow, function calls and data flow relations of a program. Previous approaches based on graph neural networks have used only graph structure (Xu et al. 20117; Ben-Nun et al.) [0188] Wang et al.) 2020). How- ever, different graph structures respond to different program features, so it is necessary to combine multiple graphs together to obtain richer structural information.
Existing methods explore program embedding through different inputs, e.g., source code, abstract syntax tree, assembly code and forth. Constructing embeddings via source code or AST makes good use of semantic information in program variables. However, they lack compile-time informa- tion and cannot solve compilation- and architecture-related analysis tasks. In order to build embed-

---

Under review as a conference paper at ICLR 2022
%1 load [32, 132 %a, align
mov
-0x10(%)rb)),
load 132, %c, align
mov
-0x1c(%rbp)
%3
load 132, %d, align
%eax, %edx
%4
load 132, 132" %a, align
-xx14(%rbp). (eerrbb)))exxxx %exx
a+)=
%6
sub nsw 132 %63 %4
%eax.%edx
%66 mul nsw [32 962. %6
mov
-Ox1c(%rbb)) %eax
%6
nsw 132 %61 %66
%eax, Seexx
%6
load 132, 132 %b, align
mov
-0x18(%rbp)
Source code
AST
IR
Assembly code
Figure 1: Different formats of program representation.
dings that can solve both source-code level tasks (program classification) and compilation-related tasks (program vulnerability analysis), we choose build embeddings through the assembly code of program. The embeddings we construct not only contain semantic and structural information of the program code, but also reflect differences in compilation configurations, so they can be applied to wider range of tasks.
In summary, this paper makes the following contributions:
We abstract program into multiple graphs, extract different program structure information, and perform multi cullt-ddimensionll analysis of the program.
We build program embedding based on the assembly code, offering the possibility to ana- lyze both source code related and compile related analysis tasks.
We perform procedure-level and program-level analysis, capturing both local and global features.
We measure program analysis tasks, program classification and binary similarity detection, and achieved an accuracy of up to 82.58% and 83.25%, 8.6% and 84.4% of improvement over the state-of-an approaches.
CHALLENGES OF PROGRAM EMBEDDING
2.1 MULTIPLE PROGRAM REPRESENTATIONS
From the perspective of language syntax, a program may be represented in many different formats, such source code, abstract syntax tree (AST), compiler IR, assembly code, etc. Fig. shows the four formats with one code example. As seen from Fig. different formats share similar information but also have their unique characteristics and complexitres. For example, only the source code and AST contain variable literals. Therefore, which level of code to embed with deserves serious study.
Program embedding from source code certainly has its own merits. A programmer uses English words with clear meaning for variable or function names when coding. Direct embedding over these names can capture the literal semanties for program analysis. Also, source code small sized, often within few kilobytes. However, the downside is having to maintain a vocabulary for all possible variable names, which can be an incredibly large number considering programmers can name variables they want. Moreover, embedding over the source code lacks information with respect to program structure semantics.
Another line of program embedding is over AST, abstract representation of program syntactic structure in the form of tree. Unfortunately, embedding over AST also suffers from the problem of oversized vocabulary lists and the shortage of program structure information.
This work chooses to embed a program based on the assembly code for several reasons. An assem- bly code be compiled from source code. It is specific processor architecture, for example x86-64. is typically less stylish and tight to program semantics. For example, programs that syntactically different but semantically equivalent tend correspond to similar assembly codes.
2

---

Under review as a conference paper at ICLR 2022
</>
CFG of function Set
Instructions of other functions
dunction Setivoldd
for to do
push dhp mov rlipusp rsp
afjiii
CG
call Sertih funetton Drmptvarii
/ffanc body""/
Block 403607
Cmpritt
Instructions of function Set
x then
(Htpovar. 4]9
Cmp
Sub
call Sertq
short 6008F2
call Dropt: function Subojitx)) x-1
Set
Drop
mov cax coxtfop-aac.
call Cmppkt:
cdqq
Z4cortv
Maint)
mov edv./fttp-vrr.f 4]
35 nop
5 then
10
mov ssafraxr4].sdk
Iorve
10
call Subiot:
11
edd (rép var.44.1
reon
22 400107
Crmppot:
Subgraph DFG
Figure 2: A CFG, CG and DFG of program.
Moreover, the assembly code often embraces more compilation-specife characteristics than the pro- gram intermediate representation or AST (L. Neamtiu Hicks 2005). This work aims to solve the task of binary similarity detection, so it is necessary to distinguish between different architectures and compilers. This compilation-relater information is not available in the source code or AST. Therefore, embedding over the assembly code can solve both source code related problems such as program classification,, and compilation--rlltee problems such as binary similarity detection.
2.2 CHALLENGES BY NUMEROUS GRAPHS OF PROGRAM
There are plenty of graph structures implicitly inside program, including but limited control flow graph (CFG), call graph (CG), and data flow graph (DFG) and more. A CFG the graphical representation of the control flow code, mostly used in static analysis. It shows all execution paths that can be traversed during program execution. Edges in CFG portray conditional branches and nodes represent basie blocks. contains loops, branches, and other structures, which difficult to obtain directly from the source code of program. A CG represents calling relationships between functions. Each node represents function and each edge (a, b) indicates that the function a calling b. Thus, cycle in the graph indicates recursive procedure calls. A DFG represents def-use chains in a program. A "def" means an assignment to a variable and a "use" a reference. A program consists of large number of ordered operations forming partial-order graph. The data flow graph reflects data dependencies between these operations.
Different graphs represent quite different program structures and semantics. Program embedding requires to embed as much semantic information as possible. How to embed these graphs together in effective and efficient way is challenging problem. Furthermore, these graphs large for the efficient graph-based deep learning models (e.g., graph neural networks) to run. As tested, the number of nodes and edges of the DFG can be high 10847 nodes and 269 10 edges. These large-scale graphs incur tediously long training time using graph neural networks.
relative
METHODOLOGG
In this section, we propose program embedding methodology that embeds multiple graph struc- tures in a program as whole from the level of assembly code. For the best practice, we adopt the deep learning models of graph neural networks (Scarselli et al.] 2009) to represent the semantics of these three types graphs. The proposed methodology has two standalone training processes consisting of two models, BERT+ and Gated GNN. We fine-tune the pretrained BERT model to derive the initial embedding of each node CFG or DFG. Since the BERT model is specifically fine-tuned for program understanding, model we use is Gated GNN (GGNN) to derive the whole embedding of the CFG, CG and DFG, as GGNN is good at memorizing long-term data dependencies between nodes.
name the model BERT+, to distinguish. The second
3

---

Under review as a conference paper at ICLR 2022
Depending the specific tasks, program analysis may be in either procedure or program level. Procedure-level analysis is usually conducted to analyze a program in the unit of procedure, there- fore for local function-level tasks. Program-level analysis is for program-level tasks. In this work, we focus on two program-level analysis tasks of program classification and binary similarity detec- tion. However, procedure-level analysis tasks such function name prediction can also be solved by the proposed model, which we will leave as future work.
For procedure-level analysis, we construct a control flow graph for each procedure and combine with instruction semantics to complete the embedding of each procedure. Then the embedding of each procedure is used to do procedure-level analysis. For program-level analysis, we represent the calling relations between different procedures by call graphs and represent the data dependencies between all variables in the program by data flow graphs. We embed every call graph and data flow graph for each procedure and combine the embeddings of all procedures to give rise to the embedding of the entire program.
3.1 GRAPH NEURAL NETWORKS
The objective of Graph Neural Network is to learn the node representation and graph represen- tation for predicting node attributes or attributes of the entire graph. Graph Neural Network (GNN) (Scarselli al. 2009) structure G' (V, E") consists of a set of node V and set of edge E. Each node y E V is annotated with an initial node embedding by æ E IR and a hidden state vector ht E Rº (hg often equals to r). A node updates its hidden state by aggregating its neighbor hidden states and its own state at the previous time step. In total, T steps of state propagation are applied onto GNN. In the t-th step, node gathers its neighbors' states
aggregation as m as shown in Eq. Then the aggregated state is combined with node v's previous state h through neural network called g. as shown in Eq.2. ʃ can be an arbitrary function, for example linear layer, representing a model with parameters 6. h g(m;; h4
m = Σ [(h';:)) (1)
(2)
= GRU b(11) (3)
(MD)EE
Gated Graph Neural Network (GGNN) (ii et al. 2016) is extension of GNN by replacing g in Eq.2 with the Gated Recurrent Unit (GRU) (Chung et al. 2014) function as shown in Eq.3. The GRU function lets node memorize history long-term dependency information, as it is good at dealing with long sequences by propagating the internal hidden state additively instead of multiplicatively.
3.2 PROCEDURE-LEVER ANALYSIS
A procedure is basic unit that implements specific function. We construct embeddings for all procedures separately. The control flow of procedures and the semantics of instructions are the key features. We construct control flow graphs for all procedures and use the semantic information of instructions as the initial embedding of nodes in the graph.
The following describes how to build procedure-level embedding from assembly code.
Assembly code is a flat profile of instructions, Ins;(i E 1. ... Each instruction Ins; is composed of a series of tokens, fy(i E I... These tokens are of many types, including operators and operands. Some instructions are compute operations with respect to register values (c.g.. ADD, SUB). Some move values between registers and memory (e.g., LOAD, STORE). Others represent conditional branch or jump to other locations (e.g., J2, JMP).
basic block (i 1... K) is composed of sequence of instructions without any control flow statements. A basic block has only one entry and one exit and may have multiple following basic blocks. When the program pointer jumps from source to target block, we connect an edge from the source to the target. Viewing the basic blocks nodes and the connections as edges, we form a CFG. For instance, for x86 direct branches, there are only two possible target blocks for a given source block, which CFG (the yellow part). The program in Fig. 2 has several procedures, and we show the CFG of Procedure Set. The loop structure of this procedure is clearly shown in the figure.
refer to the true block and false block. Fig. 2 shows example of
The semantics of instructions is crucial for program analysis. We use the semantics of each basic block to construct the initial embedding of each CFG node. We consider the instruction token
4

---

Under review as a conference paper at ICLR 2022
sequence of a basic block sentence and use the BERT (Devlin al. 2018) model to embed them. In natural language processing problems, BERT is a widely used method for producing sentence embeddings. Here, we borrow BERT to obtain the semantic relations between different tokens.
The embedding of a basic block should recognize several goals, including the affinity relation be- tween words in sentence, affinity relation between basic blocks, and implicit meaning of compila- tion configurations. To achieve the three goals, we train the pre-trained BERT model by three tasks separately. The input of the BERT model is the token sequence of basic block and output is an embedding vector. In the first task, we learn the capability of exploring the affinity relation between tokens in a basic block so as to check the missing tokens. In doing so, we mask out some tokens randomly given a token sequence and predict the masked out tokens by comparing the output embed- ding by the BERT model and the ground-truth embedding. Secondly, the model learns to recognize the affinity relation between basic blocks, i.e., predicting whether two basic blocks are adjacent in the sense that the corresponding nodes are connected by edges in CFG. For training, we randomly generate a large number of basic-block pairs. If two basic blocks are adjacent to each other, the cor- responding pair is labeled as true and otherwise false. This task allows the embedding to know the execution context of a basic block. Thirdly, the model learns the capability to check the compilation configuration by determining whether two basic blocks come from the same compilation configura- tion. As result, the embedding contains the semantic information of compilation configurations. This capability is especially useful in the task of binary similarity detection, where distinguishing between different compilation configurations is needed. For training, we randomly generate large number of basic block pairs as samples, If a pair is compiled from the same configuration, a sample labeled as true and false otherwise.
As shown in Fig. 3] once we have embedded every basic block, we can build the embedding of the whole graph, which the embedding of this procedure, through graph neural network. Once the embeddings of all procedures are obtained, we complete the procedure-level analysis.
3.3 PROGRAM-LEVE! ANALYSIS
While procedure-level analysis provides us program local semanties, how these procedures are compounded to implement highly complicated functions needs to be understood with the help of program-level analysis. In doing so, a call graph and a data flow graph that characterize the global view of program analysis are studied.
call graph reflects calling relations between procedures. Each node denotes a procedure and connected by directed edges, pointing from caller to callec. The call graph clearly provides the exccution context from one procedure to another and propagates the control and data semantics across procedure boundaries. Hence, it is more than needed analyze inter-procedural program analysis problems such as memory inefficiency detection (Chabbi & Mellor- Ccllmoyy 2012) and program slicing (Gallagher et al. [199]. As shown in Fig. 2 the procedures Main, Sub, Cmp, Set, and Drop are represented by nodes, and the edges between them indicate the calling relations.
To embed the entire call graph to form program-level embedding, we initialize the node embedding of a CFG with the procedure embedding previously generated by CFG. Then we propagate the semantics of the embeddings of every node with the GGNN graph propagation algorithm, and finally generate an embedding, denoted as Geg-
While a call graph provides control flow information, data flow graph presents program execution flows of associated data value snapshots. We construct data flow graph from the assembly code. An assembly code consists of great number of instructions. single instruction executes a single step of data processing, such as computation, data load, data store, etc. Each instruction has one or more operands. An operand is defined by one instruction such as loading from memory by the L.D instruction, computed from other variables by MOV, and used by other instructions such as ADD or storing to memory by ST.
We treat instruction as node. When variable is defined by Instruction L, and used by Instruc- tion Ig. we connect a directed edge from I, to I₁. Consequently, we obtain a data flow graph of the program. It reflects the data dependencies between instructions of the program and the "def-use" chains. Fig. shoows a graph of DFG. Each node represents an instruction, the nodes in the dashed boxes are instructions belonging the Set procedure, and the ones outside the boxes represent
5

---

Under review as a conference paper at ICLR 2022
Assembbly Inadtr Nnttions
GGNN
DFG
Program embedding
Source Code
BERT
Concat
DFG
Taskt:
program classification
BERT
Compiler
Task2:
binary similarity detection
Assembly Instruction
procedure
program embedding
embedding
Figure 3: The process of program embedding.
instructions of other procedures. Relative to CFGs and CGs. DFGs are much larger. By analyzing a small program dataset, POJ-104 (Lili Mou 2016), more than 53% of DFGs have over 512 nodes and even more than 15% can exceed 1024 nodes. Following the training methodology for generating initial embedding for basic blocks in a CFG, we construct the initial embedding of each node in a DFG using BERT by viewing the token sequence of an instruction Whether two instructions are adjacent or not is indicated by whether there is an edge between two nodes in DFG. The data flow graph is also constructed emhedding called as 9dfg-
sentence in the BERT model.
embedding through GGNN, and this
As shown in Fig. 3 construct control flow graph for each procedure and obtain its embedding. Using the embedding of each procedure and the calling relations between procedures, we obtain the first program embedding Peg- Afterwards, we construct the second program embedding 94f9 using the data flow relations in the program. Putting together these two embeddings, single final program embedding is derived for the target program (Eq.4). Finally, we use this program embedding to perform various types of program analysis tasks.
Oprog = Dense(geg g4fg) (4)
4 RESULTS
4.1 EXPERIMENTAL SETUP
Benchmark. We the POJ-104 (Lili Mou 2016) dataset as test data. is collected from Pedagogical Open Judge system. The dataset contains 104 program classes written by 500 different people. After pre-processing. programs with two compilers and four compilation configurations (GCC-02, GCC-03, LLVM-O2 and LLVM-03) and use 70% of the obtained programs as training data, 15% validation data and 15% test data.
ended up with total of 38083 programs. We compile these
Task. test the efficacy, we evaluate the proposed model on two tasks, program classification and binary similarity detection. Program classification is a commonly seen program analysis task. What specific functions program implements is one of the most important features of a program. Whether the program embedding trained by the model can accurately perform the program classifi- cation task shows whether the model can capture the key information of the program.
For each program, produce label that indicates the category of program it belongs to. The labels range from to 104, indicating each of the 104 categories of programs. Once the model has generated embeddings for each program,
softmax complete this multi-categorization task. the predicted category is consistent with the label, the sample correctly predicted. We use the prediction accuracy of all samples in the test set the measure for this task.
Another task we tested is binary similarity detection. Binary code similarity detection tasks are often applied in problems such plagiarism detection, malware detection, and vulnerability search (Xu

---

Under review as a conference paper at ICLR 2022
CFG
CG
DFG
Ins2vec
MTM
gcc
llvm
80%
Acouracy
Accuracy
60%
50%
40%
25%
20%
0%
0%
CFG
CG
DFG
Ins2vec
MTM
CFG
CG
DFG
MTM
(a) Results Compared with Ins2vec
(h) Results of different compiler
Figure 4: Classification accuracy
et al. 2017). As in the vulnerability search problem, a vulnerability in one source code may spread across hundreds more devices that have diverse hardware architectures and software platforms. In order to find issues from the same source code in different binary, we need to perform binary similarity detection. Find the binary that belongs the same vulnerability code.
In this task, we use two subtasks to help evaluate the model. The first subtask is cross configuration binary code detection. The same source code is compiled with different compilers and different compilation options. Our goal is to make sure the pairs from the same source code has higher similarity scores than pairs from different code. Same as (Yu al.] 2020a), we use the siamese framework to complete the training for the similarity determination task.
For this subtask, we use the similarity distance as a measure. Each training and testing sample consists of two binary. The model will predict whether they originate from the same source code or not. Pairs from the same source code are called positive pair, and pairs from different source codes are called negative pair. The average similarity distance of all positive pairs are called positive distance, and the average similarity distance of all negative pairs are called negative distance. The larger the difference between these two distances, the better the model's ability to differentiate. To obtain training and test data, we randomly generated 100k pair of samples. In half of the samples, the two programs compiled from the same source code with different compilation configurations. In the other half, the two programs are from different source codes.
The second subtask is compilation classification. In this task will classify the compilation options which binary belongs to. There are 4 categories in total, GCC-02, GCC-O3, LLVM-02 and L.VMM- 03. We also use softmax for this multi-classification task. The accuracy of the classification used as measure.
Implementation. The inputs our model include three graphs CFG, CG, and DFG. A binary analysis tool called angr (Shoshitaishvili et al. 2016) is utilized by taking input the binary code and constructing these graphs for every single program. To complete these two tasks, each input sample is labeled to indicate its category (category id, from 1 to 104) and how it is compiled (compilation option id, from 0 to 3). To complete the first subtask of binary similarity detection, each binary pair will have label (1 for homologous / O for heterogeneous) indicating whether they come from the same source code. We call our model Multiple Tasks Model (MTM).
We show only the values of all hyper-parameters used in the final experiments. The learning rate used 0.0001 and batch size is 32, The number of steps of CFG, CG and DFG 8, 4 and 8. The output dimension of CFG, CG and DFG are 128 2048 and 128. The hidden dimension of all three graphs is 128.
Hardware. All deep learning tasks were performed 8 Nvidia Tesla V100 GPU (Nvidia Inc. 2017) cards of the Volta architecture. Each has 5120 streaming cores, 640 tensor cores and 32GB memory capacity. The CPU host is Intel Xeon CPU 8163 2.50GHz, running Linux kernel 5.0. All the other non-deep learning tasks were run on the host.
4.2 PROGRAM CLASSIFICATION TASK
For the program classification task, we compared our model with Ins2vec NBnn-Nun et al. 2018). Ins2vec constructs a graph called XFG which incorporates data flow, and control flow information
7

---

Under review as a conference paper at ICLR 2022
negative
posilive
negative
positive
Number samples
800
Number of samples
800
800
600
400
400
200
200
0
0
60000
806A11A AAAAA
(a) Similarity distance OM
(b) Similarity distance MTM
Figure 5: Similarity distance of OM and MTM
CFG
CG
DFG
MTM
OM
CFG
DFG
MTM
OM
85.0%
95.0%
Accuracy
82.5%
Accuracy
92.5%
77.5%
87 5%
75.0%
85.0%
CFG
CG
DFG
MTM
OM
CFG
CG
DFG
MTM
OM
(a) Classification accuracy for all compilation con- figurations
(b) Accuracy of compiler classification
Figure 6: Compilation option classification accuracy
of the program. It takes the edges of the graph as the context of the nodes and uses the skip-gram model to get the embedding of each node. After that the embedding of the program is obtained with LSTM. The results of the comparison are shown in Fig. 4().. Compared with Ins2vec, we improve the accuracy by 8.6%. This shows that our model has good accuracy in solving source code related tasks.
In addition, we show the effect of each submodel in this task separately. The DFG and CG submodels represent the use of Jdfy and Jey program embedding, respectively. The CFG model indicates that only the CFG embedding of each procedure is used. The embedding of the program is the average of the embedding of all its procedures. As be from Fig. when only the control flow information of the program is utilized. only 74.38% accuracy can be obtained. When combining the control flow information and procedure call information, the accuracy is improved to 79.77%. When only the data flow information of the program is extracted to construct the embedding, accuracy of 79.93% can be achieved. From the above results, it can be summarized that added, multiple program structure information is an effective way to obtain a more accurate program embedding.
Fig. (b) shows the prediction results of the model for different compilers. The model achieves 80.41 % accuracy under GCC and 84.21% % accuracy under LLVM.
4.3 BINARY SIMILARITY DETECTION TASK
For this task, we chose the state-of-the-arr model Order Matters Model(OM) for comparison. OM generates control flow graph for each program and uses GNNs to construct embeddings.
The binary similarity detection task divided into two subtasks. In the first subtask, we need to determine whether two binaries come from the same source code. It is not easy to accomplish this task on POJ-104 dataset. Because the source code to complete the same POJ question may have a high degree of similarity, it will be more difficult to determine whether they originate from the same source code after being compiled with different compilation options. Fig. 55shows the results of our tests. can be clearly that the distribution of similarity distances of the positive pair is more
8

---

Under review as a conference paper at ICLR 2022
different from that of the negative pair under our model. The average positive and negative distance of OM is 0.5668 and 0.7272. The average positive and negative distance of MTM is 0.6394 and 0.9353. The difference between the positive distance and negative distance for OM is 0.1604, and the difference for our model is 0.2959. Our model has better differentiation capabilities.
The second subtask is the compilation classification task, which needs classify the compiled configuration of the program. With this subtask, the ability of the model to distinguish between different compilation configurations can be judged. Fig. [ shows the experimental results. When classifying among all 4 compiled categories, we obtained an accuracy of 83.25%. We also tested the ability of the model discriminate between different compilers. All samples are divided into two categories, compiled by gee and compiled by Ilvm, respectively. In this test we obtained an accuracy of 94,4% as shown in Fig. 6 (b). The above results demonstrate that our model has good differentiation ability for different compilation configurations and can successfully perform compilation- related program analysis tasks.
5 RELATED WORK
In recent years, applying neural networks onto program analysis has attracted ever- ever-increasing atten- tion (Raychev et al. [014 Hindle et al. 2016)] Allamanis et al., [016bb] Pradel & Sen 2018:] Kanade jet al.] [020] Feng et al. 2020). Token embedding for program representation using existing natural language processing (NLP) techniques serve good start. [Hindle et al. (2016) proved that code was very regular and could be modeled by statistical language models. Allamanis et al. (2016b) treated the code textual tokens and introduced an iicentionbbasee neural network to summarize functions. [Pradel& & Sen 2018) made use of the information conveyed by natural language elements in the source code, e.g., names of variables and functions, by representing them with Word2 for bug detection. Kanade et al. (2020) leveraged BERT to perform high quality source code embedding for program repair and showed that BERT is very effective in extracting the semantic information of code. Even though the above works have shown the efficacy of token embedding to some extent, they are short of structure-based semantic representation. Hu al. (2018): Chen et al. (2018) ini- tially considered ASTs based on finding that program representation is different from traditional machine translation because the code owns structures. Alon et al. (2019a a by 2020) represented a code snippet as the set of compositional paths in the AST and solved the problems of code completion, method name prediction and code documentation. There is a large body of works employing GNNs to mine program structure oriented semantics. Xu et al.] (2017) explored the embedding of binary functions out of CFGs to detect cross-platform binary similarity by measuring the distance between the embeddings of two functions. Lu et al. (2019) attempted to fuse the static AST, dynamic data flow and function calling relations into GNNs. Ben-Nun et al. (2018) used the Word2Vec model to embed each token. It is encoded with the context of new conte Xtual Flow Graph which coalesces both control flow and data flow generated from the raw LLVM IR. Shi et al. (2020) and Dufour et al.) (207)) further adds the dynamic execution state of the program to the program embedding. Though with great recent ad vannemeen,, GNNs are poor at learning from large graphs due to high cost of the underlying message-pan paaagggggannngggggge procedure. WWng et al. (2020) and [Hlllendoom al. 20199) designed new model GINN and combined GNN with RNN to obtain more comprehensive information with reduced cost. Program representation has been widely used to solve various problems, such as pro- gram repair (Wang et al. 2018: Wang & Su, 2020, Tian et al., 2020), bug detection (Dinella et al.) 2020), program classification (Lu et al. 2019] Kanade et al. 2020). cache replacement (Li & Gu
2020 Liu et al. (2020). code similarity detection (Yu et 2020a b). We use program representation
to solve the program classification problem and binary similarity detection problem.
6 CONCLUSION
To solve multiple categories of program analysis tasks, we constructed program embeddings based on the assembly code of the program. The semantics of the program source code itself as well as the features of the program compilation can be captured simultaneously. Our model takes full advantage of the multi-category graph structure information present in the program and combines with semantic information. By testing on two tasks, program classification and binary similarity detection, we obtain 82.58% and 83.25% respectively.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pp. 2091-2100.
PMLR, 2016a.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pp. 2091-2100.
PMLR, 2016b.
Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured rep- resentations of code. In International Conference Learning Representations, 2019a. URL
https::/openrewiiee wttp://wwweeeeeeteeeeeeeme (oooummiid-lllgoo0555
net
Uri Alon, Meital Zilberstein, Omer Levy. and Eran Yahav. code2vec: sentations of code. Proceedings of the ACM Programming Languages, 3(POPL): 29, 2019b.
Learning distributed repre-
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Inter- national Conference on Machine Learning, pp. 245 PMLR, 2020.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural code comprehension: A learnable representation of code semantics. arXiv preprint arXiv: (al:::8006.7335 2018.
Milind Chabbi and John Bellor-Crummey. Deadspy: A tool to pinpoint program inefficiencies. In
Proceedings of the Tenth International Symposium on Code Generation and Optimization, CGO 12, pp. 124-134, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450312066. doi: 0011445/225001622550333 URL https:://cc..rgg//
2259016.22588
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program transla- tion, 2018. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, QC, Canada, pp. 2-8, 2018.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3355, 2014. URL
httt:://arii: org/ooggo/...... abs/ / 1412. 3555
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. 2018.
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learn- ing Representations, 2020. URL htttttss:://oeeeeiiiit
net/ forum wtt..nneooouuuleeeseeeeeeeee di==
Bruno Dufour, Barbara G. Ryder, and Gary Sevitsky. Blended analysis for performance under- standing of framework-based applications. In Proceedings of the ACM/SIGSOFT International 2007.
Symposium on Software Testing and Analysis, pp.
Zhangyin Feng, Daya Guo, Duyu Tang. Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Findings, pp. 1536-1547,22
Processing:
Gallagher, B. K., Lyle, and R. J. Using program slicing in software maintenance. Software Engi- neering, IEEE Transactions on, 1991.
Vincent J Hellendoom, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International conference on learning representations, 2019.
Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the natural- ness of software. Communications of the ACM, 122-112 2016.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pp. 200 200100 IEEE, 2018.
10

---

Under review as a conference paper at ICLR 2022
J. Foster I. Neamtiu and M. Hicks. Understanding source code evolution using abstract syntax tree matching. ACM SIGSOFT Software Engineering Notes, 30:1-5, 2005.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pp- 5110-5121. PMLR, 2020.
Pengcheng Li and Yongbin Gu. Learning forward reuse distance. ArXiv, abs/2007.15859 2020.
Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. In Proceedings of ICLR 16, April 2016. http:: www.ooiio.ooooommmmmmmmmomoomo.o com/
en us www.essearch/publlccccccco wwwpppbwwiiiiiiooooooooo..o gated graph sequence neural cwwnuaal-netwrree networks
Lu Zhang Tao Wang Zhi Jin Lili Mou, Ge Li. Convolutional neural networks over tree structures for programming language processing. In Proceedings of 30th AAAI Conference on Artificial Intelligence (to appear), 2016.
Evan Zheran Liu, Milad Hashemi, Kevin Swersky, Parthasarathy Ranganathan, and Junwhan Ahn. An imitation learning approach for cache replacement. arXiv preprint arXiv:2006. 16239, 2020.
Mingming Lu, Dingwu Tan, Naixue Xiong, Zailiang Chen, and Haifeng Li. Program classi- fication using gated graph attention neural network for online programming service. CoRR, Jbs//993,00804, 2019. htL.htt:///axttti wwt.:/orgggabbbabggge 03804
Nvidia-Inc. Nvidia tesla v100 gpu architecture. http:hhtttt//tt molita-architecturee/ggo
wtt.iiaa.....ooooooooooooooo content/ wtt. pdd pdf 2017.
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Languages, 2(00PSLA):1-25, 2018.
Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Progranming Language Design and Implementation, pp. 419-428, 2014.
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1 (0(())61-80 2009.
Zhan Shi, Kevin Swersky, Daniel Tarlow, Parthasarathy Ranganathan, and Milad Hashemi. Learning execution through neural code fusion. In International Conference on Learning Representations, 2020. URL httts:::: http///eeeeveeeeee/ee/ee net.
Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Audrey Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, and Giovanni Vigna. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. In IEEE Symposium on Security and Privacy, 2016.
H. Tian, K. Liu, A. K. Kaboré, A. Koyuncu, L. Li, J. Klein, and T. F. Bissyandé. Evaluating representation learning of code changes for predicting patch correctness in program repair. In 2020 35th IEEE/ACM International Conference Automated Software Engineering (ASE), pp- 981-992, 2020.
Ke Wang and Zhendong Su. Blended, precise semantic program embeddings. PLDI 2020, pp. 121-134, New York, NY. USA, 2020. Association for Computing Machinery. ISBN 9781450376136. doi: 10. 10.1145/3385412.3388999
URL https://do:: https::/ddoo.orgee 0trg/10.11455
8385412.338598
Ke Wang, Zhendong Su, and Rishabh Singh. Dynamic neural program embeddings for pro- gram repair. In International Conference on Learning Representations, 2018. URL https:
opwnreview wttts//orumiidddeee
Yu Wang, Ke Wang, Fengjuan Gao, and Linzhang Wang. Learning semantic program embeddings with graph interval neural network. Proceedings of the ACM on Programming Languages, (0) 2020.
11

---

Under review as a conference paper at ICLR 2022
Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. Neural retwork-based graph embedding for cross cross-platform binary code similarity detection. In Proceedings of the 2017 ACM SIGSAC Conference Computer and Communications Security, pp. 363-376, 2017.
Zeping Yu, Rui Cao, Qiyi Tang, Sen Nie, Junzhou Huang, and Shi Wu. Order matters: Semantic- aware neural networks for binary code similarity detection. In Proceedings of the National Conference on Artificial Intelligence and on Innovative Applications of Artificial Intelligence, pp. 1145-1152, 2020a. URL https://aaai http:///sss..oe.ppp/aaaaaaa/aaaa milew54660
Zeping Yu, Wenxin Zheng, Jiaqi Wang, Qiyi Tang, Sen Nie, and Shi Wu, Codecmr: Cross-modal retrieval for function-level binary souree code matching. Advances in Neural Information Pro- cessing Systems, 33, 2020b.
12


Reviews:
[{"review_id": "YpNIcY6y6jF", "reviewer": "Reviewer_MnJW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\nThe paper presents an interesting idea. The usage of multiple representations together is both intuitive and promising.\n\n\nWeaknesses\n-----------------\n- The evaluation of the paper is very weak (see my detailed comments)\n- No comparison against existing ML-based binary similarity detection tools. \n- No comparison across different  hardware architectures \n- Scalabailtiy of the technique for large real-world programs is not clear \n\n\nOverall, even though the idea is interesting, the evaluation section is very weak in the current version of the paper. The authors use an old dataset from 2016 of mostly small programs with only two compilers. Usually, binary analysis tasks like similarity detection get harder with multiple compilers and optimization levels, and across multiple hardware architectures. It is fairly standard in the ML-guided binary analysis papers to evaluate against real-world programs with all such variants. Please see: \"TREX: Learning Execution Semantics from Micro-Traces for Binary Similarity\" by Pei et al. for a detailed list of related works as well as a much larger real-world dataset. I strongly encourage the authors to use such real-world large datasets, experiment across many compiler/architecture/optimization variants, and compare against different existing baselines as done by Pei et al. I also do not quite understand why figure 3 does not show binary code at all given that in many settings the corresponding source code might not be available? How is the proposed technique supposed to work in such cases?", "summary_of_the_paper": "This paper presents a graph neural network-based approach to solve two binary analysis tasks (program classification and binary similarity\ndetection). The key idea of the paper is to merge different forms of representation of binary code (compiler IR, assembly code, etc.). ", "summary_of_the_review": "The ideas are interesting but the evaluation is subpar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X1aca934657", "reviewer": "Reviewer_PoNp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Novelty\n\nThe idea of fusing feature dimensions in machine learning for code is not new. See https://arxiv.org/ftp/arxiv/papers/2109/2109.03341.pdf on composite program graphs, or https://arxiv.org/abs/1907.02136 on different types of program features.\n\n### Terminology\n\nAuthors need to be more precise about the terms they use. I know this is a learning conference, still, it is imprecise to call program classification, duplicate code detection or the other kinds of applications program analysis tasks. Program analysis in PL refers to data-flow analysis, pointer analysis, etc. if we talk about static analysis. \n\n### Others\n(1) In page 4, the Eq.1. should use u in f to represent the neighbor of v. But you used v incorrectly.   \n(2) In Figure 2, the meaning of the nodes in Subfigure of DFG is not clear, and the variables or instructions represented by each node should be specified.  \n(3) In the experimental setup, the author introduced the POJ-104 data sets, but did not introduce the code size of the program in the data set. As mentioned earlier, the DFG of the source code contains a large number of nodes and edges, but did not compare the number of nodes and edges generated by the DFG in my work.  \n(4) In the program classification task, the author proves that he can better solve semantics-based problems by comparing with Ins2vec. However, this comparison cannot reflect the advantages of doing this work on assembly code, because both of the program representation and model are different.  \n(5) On page 8, row 4, how is 8.6% calculated?  \n(6) In the program classification task, there is no explanation why the effect on gcc is worse than that in llvm.  \n(7) How to calculate similarity in binary similarity detection task?  \n(8) Clone detection is usually performed at a function or a more detailed level, and what is the practical significance of detecting the binary similarity of the entire program.  \n(9) The author has used CFG and DFG respectively, have you considered the effect of using PDG?  \n(10) The graphical representations(come from angr), bert, and ggnn used in this work are all existing works, the author just applied them to the assembly code. What is the difficulty of learning semantic information from assembly code compared to source code?  \n\n\n\n\n", "summary_of_the_paper": "This work proposes a new approach to solving semantics-based problems and compilation-related problems in program analysis by leveraging graph neural networks from the level of assembly code.", "summary_of_the_review": "Overall, I think the paper does not pass the novelty test. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3YRgq_PhbS0", "reviewer": "Reviewer_r5sH", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Sadly, I'm not convinced this approach is the right one for \"general intelligence\" on program analysis. In fact, I would argue, it's likely almost exactly the opposite of the right approach for reasoning about programs. \n\nIf we consider the recent work in the space of program reasoning, we see a definitive body of research moving away from lower levels of intermediate representations (IRs) of code (such as assembly or complier IRs, like LLVM) to richer, higher-order representations like the Intel Labs/MIT/Georgia Tech's context-aware semantics structure (CASS), Berkeley and Facebook's simplified parse tree (SPT), the contextual flow graph from Neural Code Comprehension, and even formal mechanisms such as those used by Prof. Alvin Cheung in his verified lifting work that lifts semantics from code to a higher-order lambda calculus.\n\nThe purpose of many of these higher-order representations is it can open the doorway for full algorithmic re-writes, which could result in optimizations that are outside of the scope of traditional compilers (e.g., changing the computational class of an algorithm from O(n^2) to O(n logn), etc.). These types of algorithmic re-writes have not been possible (at least from the 100s of papers I've read in the space of machine programming) using lowering techniques like the ones the authors propose.\n\nWhile I applaud the authors for investigating this approach; sadly, it has been already fairly thoroughly explored (consider Mike Carbin's work on Ithemal for a concrete example of ISA analysis and the limitations they found in it -- in short, their reach was never beyond basic blocks).\n\nI hope this review is helpful for the authors and I'm sorry that I can't give this paper a vote of confidence. I really like the area of research the authors are exploring, but their approach is not something I am convinced has novel contributions nor am I convinced that the experimental results are compelling. My apologies.", "summary_of_the_paper": "I believe the purpose of this paper is to try to understanding programs by analyzing assembly code using graph neural networks.\n\nThe authors test their system against two program-level tasks and claim an accuracy of 82.58% and 83.25%.\n", "summary_of_the_review": "TL;DR: Great area of research, but I found little to no novelty in the paper. Moreover, as a scientist, I believe this approach is likely the opposite of what we should actually be doing for program reasoning. That is we should be lifting program semantics *upwards* to understand their meaning, not lowering programs *downwards* to understand their meaning.\n\nReject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "291416-tJA4", "reviewer": "Reviewer_Cbj6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+Addresses a challenging problem of building a model to achieve better performances in both compilation-specific and source-code level program analysis tasks.\n\n+Designs a method for generating procedure and program level embeddings from multiple graphs.\n\nConcerns:\n\n1. Novelty of the Modeling Approach: The proposed method of generating the procedure-level embeddings of a program including the use of BERT for embedding basic blocks in the CFGs is very similar to [2]. Although the experimental section compares against [2], an attribution to [2] in the methodology would make the contribution of this work clear to the reader. The contribution of the proposed modeling approach seems to be the addition of CG and DFG for program-level analysis, which could be considered as very incremental. \n\n2. Weak Baseline Used for Source-Code Level Tasks: Different from [2], this work proposes a model that performs both source-code and compilation-related tasks. However, a very relevant work [1] that also uses GGNN to model a program graph (based on CFG and DFG) and targets source-code level tasks is not discussed. Any improvements in the proposed approach (by using assembly code and multiple graphs) should be evaluated against [1] or similar work instead of Ins2vec, which I think is not the strongest (or the most relevant) existing approach. Overall, I think a comparative analysis with [1] or similar work is required for establishing the generalizability claim of this work.\n\nQuestions/Comments for the Authors:\n1. The methodology of pre-training BERT looks very similar to [2], which performs the training only on CFG.  The proposed approach uses three of the four tasks used in [2]. The task not included is “the block inside graph task (BIG)”, which was effective for the model to understand the relationship between block and graph. Is there any specific reason for not using this task? \n2. Should we expect a similar gain in performance for procedure-level analysis tasks? Abstracting a program into multiple graphs: CFG, CG, and DFG derived from assembly code may not provide additional information for procedure-level tasks like function name prediction compared to existing methods (e.g., [1]).\n3. In Figure 6, it seems the accuracy of OM [2] is lower than the CFG submodel. What are the factors contributing to this change in performance? \n\nMinor Comment:\n\nAllamanis et al. 2016a and Allamanis et al., 2016b are duplicate entries of the same work.\n\nReferences:\n\n[1] Allamanis, Miltiadis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. ICLR, 2018.\n\n[2] Zeping Yu, Rui Cao, Qiyi Tang, Sen Nie, Junzhou Huang, and Shi Wu. Order matters: Semantic- aware neural networks for binary code similarity detection. In Proceedings of the National Conference on Artificial Intelligence and on Innovative Applications of Artificial Intelligence, pp. 1145–1152, 2020\n", "summary_of_the_paper": "This paper proposes a program analysis model based on graph neural networks that performs analysis on the assembly code of a program and uses Control Flow Graph (CFG), Call Graph (CG), and Data Flow Graph (DFG) of the program as inputs. The goal is to design a generalized model that can solve both source-code level tasks (e.g., program classification) and compilation-related task (e.g., vulnerability analysis). Program embedding based on the assembly code allows the model to learn compilation-specific features and the use of multiple graphs reflect semantic and structural information.", "summary_of_the_review": "Contributions are not well-justified. A very relevant work that addresses similar challenges and applies similar approach is not discussed. Demonstration of improvements are shown comparing to a weak baseline instead of more advance and relevant existing works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to neural analysis of programs whose main feature is that it operates on assembly code, therefore can account for issues that depend on things like compiler settings. The other important claim of the paper is that by combining information from the control flow and data-flow graphs extracted from the assembly code, they are able to produce an embedding that can support a variety of tasks. 

The meta-reviewer agrees with the reviewers that this paper is not suitable for acceptance at ICLR. The novelty in the approach is quite limited, and the gap between the extremely bold claims of the introduction of the paper and what is actually proven in the experiments is quite significant. The evaluation is not strong enough to be considered state of the art.
Decision: Reject