CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code
Authors: Tong Ye
Lingfei Wu
Tengfei Ma
Xuhong Zhang
Yangkai Du
Peiyu Liu
Shouling Ji
Wenhai Wang
Venue: EMNLP 2023 Main (2023)
Keywords: binary code summarization

Abstract: Automatically generating function summaries for binaries is an extremely valuable but challenging task, since it involves translating the execution behavior and semantics of the low-level language (assembly code) into human-readable natural language. However, most current works on understanding assembly code are oriented towards generating function names, which involve numerous abbreviations that make them still confusing. To bridge this gap, we focus on generating complete summaries for binary functions, especially for stripped binary (no symbol table and debug information in reality). To fully exploit the semantics of assembly code, we present a control flow graph and pseudo code guided binary code summarization framework called CP-BCS. CP-BCS utilizes a bidirectional instruction-level control flow graph and pseudo code that incorporates expert knowledge to learn the comprehensive binary function execution behavior and logic semantics. We evaluate CP-BCS on 3 different binary optimization levels (O1, O2, and O3) for 3 different computer architectures (X86, X64, and ARM). The evaluation results demonstrate CP-BCS is superior and significantly improves the efficiency of reverse engineering.

CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code
Tong Ye!, Lingfei Wu?, Tengfei Ma², Xuhong Zhang', Yangkai Du
Peiyu Liu', Shouling Ji', Wenhai Wang';
'Zhejiang University; *Anytime.Al; SStony Brook University
{tongye, zhangxuhong, yaw.ggguuooonggggiiiiii.i yangkaidu, wwwpeliuppuusscemmo y zdzzlabjzzzzzzzzz edu.cn Iwu@anytime-ate com, tengfei www.stonyboookkcooomo edu
Abstract
with limited high-level information, making it dif- ficult to read and understand, as shown in Figure Even an experienced reverse engineer needs to spend a significant amount of time determining the functionality of an assembly code snippet.
Automatically generating function summaries for binaries is an extremely valuable but chal- lenging task, since it involves translating the ex- ecution behavior and semantics of the low-level language (assembly code) into human-rees readable natural language. However, most current works on understanding assembly code are oriented towards generating function names, which in- volve numerous abbreviations that make them confusing. To bridge this gap, we focus generating complete summaries for binary functions, especially for stripped binary (no symbol table and debug information in real- ity). To fully exploit the semantics of assem- bly code, we present a control flow graph and pseudo code guided binary code summariza- tion framework called CP-BCS. CP-BCS uti- lizes a bidirectional instruction-level control flow graph and pseudo code that incorporates expert knowledge to learn the comprehensive binary function execution behavior and logic semantics. We evaluate CP-BCS on different binary optimization levels (01, 02, and O3) for 3 different computer architectures (X86, X64, and ARM). The evaluation results demonstrate CP-BCS is superior and significantly improves the efficiency of reverse engineering.
1. mov [xdi]
rax,
Jz short loc FT1.919788175220
push zbp
rbp, rap push rbx rbx, rdi
mov rdi, [rax+8]
rdi, rdi
10. jz short loc F
12.
mov rax, [rax+48h]
13.
call
ptz [zax+28h]
14.
mov rax, [xbx]
15.
rdi,
16.
call sub ptt___r11111122220000
rdi, perF1rr98114768
18.
call sub batb__eeeeee
19.
mov qword ptr [xbx]
20.
mov eax,
21.
pop
22.
pop xbp
23.
Figure I: sample of assembly code. The function
name is: gss pss_del sel_see__oo_ sec_context. The summary is: free all resources associated with context handle.
1 Introduction
To mitigate this issue, researchers have made initial attempts, with recent studies focusing on pre- dicting function names of binaries (Gao al., 2021; Jin al., 2022; Patrick-Evann et al., 2023). Func- tion name prediction is the process of automatically generating function name for given assembly code snippet, which aims at showing the high-level meaning of the function. As shown in Fig 1, the target function name is "gss_det cess_ddl_sece _______cccc_c"..... Al- though some progress has been achieved in func- tion name prediction, function names themselves can only partially and superficially represent the semantics of assembly code. Furthermore, function names frequently contain various abbreviations and custom tokens defined by developers (c.g., "gss", " "del", "sec" in the example above). Consequently,
Most commercial off-the-shelf software is closed- source and typically distributed as stripped binaries that lack a symbol table or any debug information (e.g., variable names, function names). This prac- tice is mainly done for easy distribution, copyright protection, and malicious evasion. Professionals seeking to analyze these stripped binaries must per- form reverse enginecring and inspect the logic at the binary level. While current binary disassem- blers, such IDA Pro (Hex-Rays, 2021) and Ret- Dec (Avast Software, 2021), can translate machine code into assembly code, the assembly representa- tion only consists of plain instruction mnemonics
Corresponding author.

---

relying solely on function names can make it diffi- cult to obtain an accurate description of an assem- bly code snippet and may even cause confusion.
Code) are available. Although many of the result- ing pseudo codes are imprecise and usually cannot be compiled, it still encompasses expert knowl- edge and understanding derived from human re- verse engineers. Furthermore, considering the lack of meaningful high-level strings in pseudo code in realistic scenarios, inspired by pre-trained models such as CodeT5's (Wang al., 2021) great per- formance in source code-related tasks, we explore the potential of utilizing such pre-trained models to recover missing semantic strings in pseudo code on stripped binaries. Our objective is to further narrow the gap between pseudo code and natural language by leveraging the capability of pre-trained models, To facilitate further research in this area, we have made our dataset and code publicly available ! In summary, the contributions of this paper can be outlined follows:
We argue that generating a high Sugh-iuality descrip- tive summary is a more direct and fundamental ap- proach to strike at the essence compared to function name prediction (e.g.. "free all resources associ- ated with context "ontext_handle." in the Figure 1). Similar tasks have been extensively studied at the source code level (known source code summarization) for languages such Python and Java (LeClair al., 2020; Shi et al., 2021; Wu et al., 2021: Guo et al., 2022c). At the source code level, researchers typically rely on advanced code analysis tools to extract fine-grained code structure properties and leverage the high-level semantic information inher- ent in the source code itself. However, assembly code, low-level language, often lacks high-level, human-readable information and is prone to ambi- guity. Moreover, the absence of fine-grained assem- bly code analysis tools makes more challenging to gain a semantic understanding of assembly code. Furthermore, we discover that current large lan- guage models, such as ChatGPT (OpenAI, 2022). generally possess only a rudimentary understand- ing of assembly code without high-level abstract semantic comprehension (shown in Appendix A).
To the best of our knowledge, CP-BCS is the first system for practically stripped binary code summarization. CP-BCS fully learns the execution behavior and semantics preserved in binary functions from three perspectives.
We manually construct comprehensive dataset, which is the first dataset include {assembly code, summary) pairs for three dif- ferent computer architectures (X86, X64, and ARM) and three different optimization levels (O1, 02, and 03).
In this paper, we specifically concentrate on binary code summarization in stripped scenarios, which is highly practical setting, and present CP- BCS. The novel CP-BCS framework comprehen- sively represents the execution behavior and seman- ties of assembly code from three different perspec- tives inspired by observing how human engineers analyze assembly code in practice.
We conduct extensive experiments to evalu- ate the effectiveness of CP-BCS. The results on both automatic metrics and human evalua- tion demonstrate the superiority of CP-BCS. In particular, the human evaluation indicates that CP-BCS can significantly improve the ef- ficiency of reverse engineers' comprehension of binary functions.
(1) Assem- bly Instruction: the assembly instructions them- selves provide certain features such memory operations and setting of register values. In addi- tion, we also take into account some meaningful strings that remained in stripped binaries, such as the name of externally called function. (2) Con- trol Flow Graph: to obtain the logical execution order of the assembly code, we extract the Control Flow Graph (CFG) of the assembly code. Con- sidering the order relationship between adjacency instruction, augment the original basic-block level CFG Bidirectional Instruction-level Con- trol Flow Graph (BI-CFG). (3) Pseudo Code: due to the difficulty of understanding assembly code, plugins that attempt to decompile assembly code into high-level, C-like language (known as Pseudo
2 Related Works
Function Name Prediction in Binary. Function name prediction is a task for binaries aimed at generating binary function names. NFRE (Gao et al., 2021) proposes two data-prepioc ap- proaches mitigate the ambiguity of function names. SYMLM (Jin et al., 2022) proposes neural architecture by learning context-sensitive behavior- aware code embedding. However, they still have
htttps://githbbe httm//togyee9/Binaroodedeeee

---

OFG
BI-CFG
ner pay, [xdi]
shoct 51752004
nea, Iree]
Feed Network
vill, [eae)
push tha
01752070
BI-CFG Multi-Head
tha,,
nov [naer|]
prg rie
BI-CFG Encoder
Crose-Attention
017520022
prpp sapp
Assembly
Construction
les11
mov rax [rdá] test String Festure
last vau,
0100109
Disassemble
Assembly code Multi-Head Cross-Attention
0010100
stp.
Assembly Instruction Encoder
prpp rie
Decompilation
Pscudo
Pscudo
Faesoall www.r tioccar ppeao nail
Fine-tuned CodeTS
Multi-Head Cross Attention
LOORGED +1ma2:
19a1
vLEL)
Refined Pseudo Code
(roood (+1) (eoid) (+1 LONED
Masked
Pseudo Code Encoder
Figure 2: The overall architecture of CP-BCS.
not solved the ambiguous function name issues. XFL (Patrick-Fvee Evans et al., 2023) performs multi- label classification and learns an XML model to predict common tokens found in the names of func- tions from C binaries in Debian. As XFL predicts labels instead of whole function names, it is able to predict names for functions even when no func- tion of that name is contained in the training set. The biggest difference between them and us is that we directly generate function summary sentences rather than a few function name tokens.
human- nature of the code, preservation of information, and availability of tools. Binary code summarization, on the other hand, is more challenging due its lower-level representation, loss of information, and ambiguity.
Methodology
3.1 Overview
Our proposed CP-BCS framework is designed as a plugin in the disassembler, such as IDA Pro or an online service?, which automatically gen- erates a human-readable descriptive summary for stripped functions. The whole architecture of CP- BCS is presented in Figure 2. As prerequisite, the stripped binary is disassembled into assembly code by IDA Pro, and the functions in the binary are correctly recognized. The assembly code is then input into CP-BCS, which is essentially an decoder architecture, to ultimately gener- ate the corresponding summary. CP-BCS consists of three encoders (Assembly Instruction Encoder, BI-CFG Encoder, and Pseudo Code Encoder) and summary decoder. Next, elaborate the principle and implementation of CP-BCS.
Source Code Summarization. Both binary and source code summarization aim to generate con- cise and human-readable summary of given code snippet. However, there are many sophisticated tools available for source code, such as parsers and token-level code analysis tools, which can help with the summarization process. Based on these tools, many approaches propose exploiting source code's s structural properties, including Ab- stract Syntax Tree, Program Dependency Graph in a hybrid way (Iyer et al., 2020; Choi et al., 2021; Shi et al., 2021; Zhu et al., 2022), or structured- guided way (Son et al., 2022; Guo et al., 2022e; Ye et al., 2023). In contrast, binary code analy- sis tools are much coarser and can only achieve basic functionalities such as block jumping and function cross-referree referennffrn...gg.. summary, source code summarization is generally easier due to the
3.2 Assembly Instruction Encoder
To understand the semantics of binary functions. the assembly code itself is the first-hand source that

---

can be utilized. It composes of a series of instruc- tions, each of which is responsible for performing an action, such as reading and writing register or memory addresses. Each instruction is composed of an opcode (e.g.. mov, add) and one or more operands (c.g., rax, [rdi]). We treat each opcode and operand as separate token. This is because each opcode operand carries its own semantic information, and we aim to learn the semantics of each word as finely as possible rather than treating the entire instruction as one token, like in binary function name prediction (Gao et al., 2021).
Further, traditional CFGs are unidirectional, which means each instruction cannot receive informa- tion from the instruction executed after it. To ad- dress these limitations, we propose a Bidirectional Instruction-level Control Flow Graph (BI-CFG). BI-CFG treats each instruction as node and incor- porates the logical execution order between instruc- tions, well as the jumps control flow between basic blocks, achieving level of granularity the instruction level. Furthermore, BI-CFG allows each instruction aggregate node features from both forward and backward instructions, enabling bidirectional processing.
Although stripped binaries lack symbol tables and debugging information, we have found that there is still some string information in the assem- bly code, such the names of externally called functions, which called string features. These string features provide additional high-level infor- mation that can help to some extent in understand- ing the behavior of the assembly code.
To improve the representation ability of BI-CFG, advanced graph neural networks are adopted to achieve this goal. Taking advantage of the GAT's (Veličković et al., 2018) exceptional performance and its ability to assign adaptive attention weights to different nodes, we employ GAT Encoder (GA- TEnc) to represent each node in the BI-CFG. The GATEne layer processes the BI-CFG by first aggre- gating the neighbors of the instruction nodes with edge information. It then updates the instruction nodes with the aggregated information from their neighborhoods. After updating the node informa- tion, the node representations are put together into LU activation followed by residual connec- tion (He et al., 201 and layer normalization (Ba et al., 2016). Assuming the BI-CFG contains g in- struction nodes [m. 17,,77,,- n2.,,,, nzgl, after the GATEnc, each node has a semantic representation:
We input the assembly tokens and features into the Assembly Instruction Encoder (AlEnc). The AlEnc is essentially a Transformer encoder (Vaswani et al., 2017), which consists of stacked multi-head attention and parameterized linear trans- formation layers. Each layer emphasizes on self- attention mechanism. Considering that the seman- tic representation of the opcode and operand does not rely on the absolute positions, instcad, their mutual interactions influence the meaning of the assembly code. To achieve this, we adopt a rela- tive position encoding (Shaw et al., 2018) instead of an absolute position to better learn the seman- tie representation of each assembly token. The assembly code snippet is assumed consist of p tokens [t,. f2...., ..., tpl. after AlEne, each token has a corresponding semantic representation, which is denoted as:
[M1, T2, Tyl G.T GnT'Enc([1, .2,,... my]))
3.4 Pseudo Code Encoder
Considering that assembly code is extremely low- level and hard to comprehend, there is a large gap between it and natural language summary. How- ever, plugins available that can facilitate the comprehension of assembly code by decompiling into pseudo code. Compared assembly code, pseudo code is a higher-level C-like language and can narrow the gap and alleviate the difficulty for reverse engineers to analyze assembly code. Al- though the generated pseudo code is not precise and often cannot be compiled, it still embodies ex- pertise and comprehension derived from human verse engineers. We believe that integrating pseudo code with expert knowledge can facilitate more comprehensive comprehension of the semantics of assembly code from an alternative perspective. However, in real-world stripped scenarios,
[hi, h2 he,, hpl 12 12,., 4D)
3.3 BI-CFG Encoder
In order to better understand the structure and exe- cution behavior of assembly code, we extract the Control Flow Graph. A canonical CFG is com- prised of basic blocks and jump control flows. The nodes portray basic blocks, and the edges portray jump control flows, as shown in the upper left cor- ner of Figure 2. However, it should be noted that canonical CFGs are based on basic blocks, which overlook the sequential execution relationships be- tween adjacent instructions within basic blocks.

---

pseudo code often lacks meaningful strings, such as variable and function names, which are replaced by placeholders. This inspires us to explore ways to recover these missing strings as much as possible. With the emergence of pre-trained models, such as CodeTS (Wang et al., 2021), Unixcoder (Guo et al., 2022a), which have demonstrated remark- able performance source code-related tasks, are motivated consider utilizing the pre-trained models' comprehension source code and natural language recover the missing semantic strings in pseudo code to the fullest extent possible. To achieve this goal, we take the pseudo code decom- piled from the stripped binary as input and the cor- responding pseudo code decompiled from the non- stripped binary as the target to fine-tune C5 shown in Figure 3. We expect the fine- tunedtuned CodeT5 can recover meaningful strings in the orig- inal pseudo code, such as function names, variable names, and other comments, etc. Following such recovery, the original pseudo code is enriched with more high-level string content, which we refer to refined pseudo code.
Santoala
Lenas ruts /f Sas
Pscudo Code stripped binary
catau 5e800021A.
vL.23
pott 1955 em13)5
nat SEA.
CodeT5
Jnd84 fastcall
geeaa mut. eaa
45. real
Pscudo Code in non-stripped binary
Jaceon descconss.
(*troid roosd) en C,000 +)
22133 s11A)
(ral) ral ESS;
Figure 3: Fine-tune CodeT5 using Pscudo Code from stripped binary and corresponding Pseudo Code from non-stripped binary.
into a feed-forward network for non-linear transfor- mation.
4 Dataset Construction and Statistics
4.1 Dataset Construction
For refined pseudo code, we employ an addi- tional encoder, known Pscudo Code Encoder (PSEnc), that is identical the AlEne for repre- sentation learning. Assuming the refined pseudo code contains tokens PP..PP.....PP Pn), after PSEnc, each token has a semantic representation, which is denoted as:
It is non-trivial to obtain high-quality datasets for binary code summarization in the stripped scenario. The construction process of the entire dataset is shown in Figure 4.
L Preliminary Survey. We conduct prelimi- nary investigation with 15 reverse engineers from academia and industry to explore the types of bina- ries that reverse engineers encounter in their daily work, as well as other related questions (further details can be found in Appendix B). Additionally, we also include binaries commonly utilized in other binary-related tasks, such as binary clone detection (Ding et al., 2019; Yang et al.. 2022). In total, we identify 51 corresponding binary projects in real- world scenarios. The specific binary projects are listed in Appendix
(U1, U2,
....... ... mn] PSEnc([Pn, Pp.. Pnl)
3.5 Summary Decoder
The summary decoder designed with modified Transformer decoding blocks. At time step f, given the existing summary tokens [$1, s2, 81-1), the decoding blocks first encode them by masked multi- head attention. After that, expand the Trans- former block by leveraging three multi-head cross- attention modules interact with the three en- coders for summary decoding, as shown on the right side in Figure 2. A multi-head cross-attention module is applied to the pseudo code token fea- tures to obtain the first-stage decoded representa- tion. This representation is then passed through another multi-head cross-attention module over the learned assembly token features for the second- stage decoding, which is further fed into the third multi-head cross-attention module over the learned instruction node features for the third-stage decod- ing. Then the decoded summary vectors are put
@ Source Code Collection. Based on the prelim- inary survey, collect these 51 binary projects and their corresponding source code from Github or their official websites.
S Compiled Source Code. We manually com- pile these binary projects using the compiler (gec- 7.3.0) into three different optimization levels (O1, O2, 03) for three different computer architectures (X86, X64, ARM). noted that each binary file contains nine different variants.

---

① Making of Pairs. Initially, we use the func- tion boundaries as indices to assign the function name to the function in the stripped binary. Next, we use function_ nane as indices to connect the summary and the corresponding stripped assembly code together. Finally, we construct pairs in the for- mat of (stripped assembly code, summary), which forms instances of the final Dataset.
Prellminary Survey
Binary Projects Collection
Source Code
Binary
Stripped Binary
Summary
4.2
Dataset Statisties
5
Datasets (Arch: X64)
01
02
03
Dataset
Assembly Code
Train
12,801
11,949
10,812
Validation
1,600
1,494
1,351
Test
1,599
1,493
1,351
Figure 4: The construction process the dataset.
Assembly Code: Avg. tokens
213.08
222.61
316.47
BI-CTG: Avg. modes
42.57
44.83
57.54
BI-CFG: Avg. edges
62.78
69.14
93,84
+ Summary Extraction. We extract separate function-summin summary pairs from the source code. Spe- cially, we extract functions and the associated com- ments marked by special characters "y**" and "**"" over the function declaration. These comments can be considered as explanations of the functions. We filter comments inside the function, and the first sentence was selected as the summary, which is consistent with the approach used in extracting summary in the source code summarization domain (Hu et al., 201 8a; Liu al., 2021). As a result, we get (function_nande name, summary) tuples.
Pseudo Code: Avg. tokens
228.65
243.32
359.99
Summary: Avg. tokens
9.74
9.58
9.68
Table I: Dataset statistics for X64 architecture with (01, O2, and O3) optimization levels.
Table 1 displays the statistics of three datasets under three optimization levels on the X64 archi- tecture. Each specific architecture and optimization level corresponds to a specific dataset. The statis- ties of the datasets for two other architectures (X86, ARM) and some additional explanations about the datasets can be found in Appendix D.
5 Binary Stripping. To ensure consistency with the real stripped scenario, employ the "strip -s" command strip the binary. The strip opera- tion removes sections such as "debug". "symtable". "strtab", etc., resulting in the elimination of sym- bol tables and all debugging information from the binary file.
Experiments
5.1 Experimental Setup
Out-of-Vocabulary The vast operators in assem- bly code may produce a much larger vocabulary than natural language, which can cause Out-of- Vocabulary problem. To avoid this problem, in- spired by related studies (Gao et al., 2021; Patrick- Evans et al., 2023), we empirically set the follow- ing rules to normalize assembly code:
@ Binary Disassembling. We use IDA Pro (Hex- Rays, 2021) to disassemble the original binary and the stripped binary to obtain their corresponding as- sembly code. We then separate the assembly code at the function level. For the assembly code from the original binary, we extract tuples in the form of {fimction_naane _ane, function ). How- ever, in the stripped binary, the function name is replaced by placcholder sub_ sab_address, but the function boundaries remain unchanged whether the binary is stripped. For the assembly code from the stripped binary, we extract triplets in the form of liibb__ddres, stripped assembly code, function boondariisnss
Retaining all the mnemonics and registers.
Replacing all the constant values with <Posi- tive>, <Negative> and <Zero>.
Replacing all internal functions with <ICall>,
Replacing all the destinations of local jump with <JumpAddress>.
Metries. Similar source code summarization. we evaluate the binary code summarization perfor- mance using three widely-used metrics, BLEU (Pa- pineni et al., 2002), METEOR (Banerjee and Lavie,

---

2005) and ROUGE-L (Lin, 2004). Furthermore, to provide more accurate reflection of actual per- formance, we have designed human evaluation that includes three aspects: Similarity (the sim- ilarity between CP-BCS generated summary and the ground-truth) Fluency (the fluency level of the results generated by CP-BCS) and Time-Cost (to what extent our model can improve the efficiency of reverse engineering). Further details on the human evaluation are deferred Appendix E.
2.43 and 2.83 BLBU points, respectively. This is attributed to the simpler and more flexible Reduced Instruction Set Computing (RISC) architecture of ARM, while X86 and X64 rely on the Complex Instruction Set Computing (CISC) with a larger number of operation codes and registers to support complex mathematical operations, making it more challenging for CP-BCS to understand their assem- bly codes.
(2) CP-BCS performs better under the O1 optimization level compared to O2 and O3. Through our empirical observation of assem- bly code under different optimization levels, the O2 and O3 optimization levels employ abundant advanced techniques such as vectorization instruc- tions and loop unrolling to improve program exe- cution speed but generate more complex assembly code. By contrast, Ol uses simpler methods, such as register allocation and basic block reordering. without generating overly complex assembly code, which can also be reflected in dataset statistics Table Thus, the assembly code generated by 01 is relatively simpler and easier for CP-BCS to extract semantic features.
Training Details. We implement our approach based on NVIDIA 3090. The batch size is set to 32 and Adam optimizer is used with an initial learning rate 10-4. The training process will terminate af- ter 100 epochs or stop early if the performance on validation set does not improve for 10 epochs. In addition, we leverage greedy search during valida- tion and beam search (Koehn, 2004) during model inference and set beam width to 4.
5.2
Main Results
ARCH
OPT
BLEU
ROUGL-L
METEOR
ARM
01
29.75
27.84
16.81
5.3
Baselines and Ablation Study
ARM
02
29.56
27.67
15.98
ARM
03
26.66
24.26
14.03
Avg.
28.66
26.59
15.61
Model
BLEU
ROUGL-L
METEOR
Assembly Code Only
22.88
18.82
11.09
X86
01
26.57
25.04
13.50
Pseudo Code (CodcT5)
22.89
22.04
11.89
X86
02
25.74
23.74
13.34
Pscudo Code (CodeTS+)
24.14
23.83
12.48
X86
03
26.38
25.04
13.24
Pscudo Code (UniXcoder)
23.17
22.65
12.35
Avg.
26.23
24.60
13.36
CP-BCS w/o Pseudo Code
24.50
21.54
12.35
X64
01
26.86
26.62
14.59
CP-BCS w/o BI-CFG
24.37
21.75
12.53
X64
02
25.50
23.64
12.70
CP-BCS w/o Refined
25.61
23.20
13.12
X64
03
25.14
23.92
13.30
CP-BCS (Full Model)
26.86
26.62
14.59
Avg.
25.83
24.73
13.53
Table 3: Baselines and ablation study results on the
Table 2: CP-BCS overall performance across different architectures (ARCH) and optimizations (OPT).
dataset for X64 architecture with O1 optimization level.
While binary function name predic- tion methods exist and have made processes, such as mitigating the ambiguity of function names (Gao et al., 2021) and converting to multi-label classi- fication (Patrick-Evans et al., 2023), their entire workflow and goals differ greatly from our task. Therefore, it is difficult to directly compare the per- formance of these methods with our approach. We adopt "Assembly Code Only" and "Pseudo Code" as our baselines. The formal solely uses assembly code to generate the summary, while the latter uses the corresponding pscudo code and summary pairs to fine-tune pre-trained models, such as T5
Baselines.
We first evaluate the overall performance of CP- BCS on our datasets. As shown in Table 2, the BLEU metric falls within the range of 20-30, in- dicating that "the gist is clear, but has grammati- cal errors" according Google interpretation' of BLEU, Besides, there are two interesting findings: (1) CP-BCS performs better on the ARM archi- tecture compared to X86 and X64. On average, CP-BCS on ARM outperforms X86 and X64 by
chttps://clloe google. con/ ttanssatteeatoolllll docs/evaluate. Intervals of 10-19 indicate that the summary is "hard to get the gist", while intervals of 30-40 mean the summary is "understandable to good translations".

---

(Wang et al., 2021), CodeT5+ (Wang et al., 2023) and UniXcoder (Guo et al., 2022b). We select these two classes as baselines because they are the most straightforward and intuitive ways to tackle the task.
We conduct human evaluation (details pro- vided in Appendix E) to assess the quality of the generated summaries by CP-BCS in terms of Sim- ilarity, Fluency, and Time-Cost, as depicted in Figure 5. The results on the similarity and fluency metrics show that CP-BCS, can generate summaries that are more similar to the ground truth and more fluent in naturalness. Moreover, the time-cost sults indicate that CP-BCS significantly enhances the efficiency of reverse engineers' comprehension of assembly code. In particular, compared the "None" scenario (only given assembly code), CP- BCS improves speed by 9.7 times.
Ablation Study. To evaluate the effectiveness of CP-BCS components, conduct set of ablation studies. We design three models for comparison, each one removing an important component from CP-BCS, as follows: (1) remove BI-CFG, labeled as CP-BCS w/o BI-CFG; (2) remove pscudo code, labeled as CP-BCS w/o Pseudo Code; (3) keep pseudo code but without refind, labeled as CP- BCS w/o Refined. For demonstration purposes, we choose dataset with a specific architecture (X64) and optimization level (01). The ablation experi- ment results for other architectures and other op- timization levels (the remaining eight groups) are included in Appendix F. As shown in Table 3, the performance of EP-BCS is affected when any of these components are removed. The result of CP- BCS w/o BI- BI-CFG and CP-BCS w/o Pseudo Code show that the B1-CFG and pscudo code are the most significant learning components of CP-BCS. Re- moving BI-CFG and pseudo code resulted in per- formance decrease of 2.49 and 2.36 BLEU points, respectively. Moreover, the performance of CP- BCS w/ Refined indicates that refined pseudo code can further enhance the performance of CP-BCS; a detailed case is shown in Section 5.6. Similar conclusions can be drawn from the ablation experi- ments other datasets, further demonstrating the universality of the three important components.
5.5 Study on the Model Structures
In this section, evaluate the performance of CP- BCS across varied model structures. Specially, we investigate the impact of the sequencing among three distinct cross-attention modules the sum- mary decoder on the final performance. Further- more, we explore the implications of directly con- catenating assembly code with pseudo code and using single encoder for representation.
Crussutttention Module Orders assembly code-BI CIG -pseudo code assembly code spseddo cade CBL
BLEU
ROUGL-L
METEOR
26.71
26.37
14.45
26.50
26.40
14.39
BLCCFG vassembly code spoendo code
26.45
25.95
14.31
BI- CF-CFGG-uuuooo cede codeemysseblly code
26.47
26.08
14,49
pseudo code -assembly cude CTG
26.86
26.62
14.59
pseudo code BBLCPG sassemblyy code
26,86
26.45
14.67
Table 4: Different cross-attention module orders on the dataset for X64 architecture with O1 optimization level.
ARCH:X64; OPT:01
BLEU
ROUGL-E
METEOR
5.4 Human Evaluation
concat (assembly + pseudo)
24.49
21.71
12.68
concar (assembly pseudo) BL-CFG
25.83
24.33
13.55
CP-BCS
26.86
26.62
14.59
Assembly Only
Pasudo Code
CP-BCS
None
Function Namt
Table 5: Directly concatenation of assembly code and pseudo code on the dataset for X64 architecture with O1 optimization level.
4.37
87X
181
Evalurtion
3.41
6.5X
184
Table 4 presents the performance of different orders sirst-xsecod--o ->thrrd) among the three dis- tinct cross-attention modules (assembly code, BI- CFG, and pseudo code) in the summary decoder on the dataset for X64 architecture with O1 optimiza- tion level. The results shows that different orders only have a slight impact on the final performance (the BLEU score did not fluctuate by more than 0.5 points). In Table 5, we use "concat (assem- bly + pseudo)" to present directly concatenating
111
2
1,27
2.2X
0
1X
Similarity
Fluency
Time-Cost
Figure 5: Human evaluation. "Assembly Code Only" and "Pseudo Code (CodeT5+)) are the two baselines. "None" means only given assembly code; "Function Name" means given assembly code and the correspond- ing function name.

---

assembly code with pseudo code. The results show that using single encoder represent the concate- nated body of assembly code and pseudo code can degrade the model's final performance. Therefore, assigning a separate encoder for assembly code and pseudo code is a better choice.
Limitations
Although our approach has been proven effective, it does not take into account code obfuscation (Men- guy et al., 2021; Schloegel et al., 2022). Code ob- fuscation is a technique that alters the structure and logic of program' code to make it difficult to an- alyze, preventing malicious actors from obtaining sensitive information exploiting its vulnerabil- ities. We treat code obfuscation as an orthogonal problem, and any progress made in addressing it would be complementary to our approach.
5.6 Case Study of Refined Pseudo Code
int -_fastcall
sub_E6D18(_DWORT +a1) {
Pseudo Code (Stripped)
if (a1[55] != dword_162354)
return 0; sub [bbbE664C(a1);
Acknowledgements
162wod__62354;
return 1;
This work was partly supported by NSFC under No.62102360, CNKLSTISS, the Fundamental Re- search Funds for the Central Universities (Zhejiang University NGICS Platform), and the advanced computing resources provided by the Supercom- puting Center of Hangzhou City University.
int -_fastcall
burn_drive_frre_sut
(burn_drive *d) {
Refined Pseudo Code
if (d->sub. nwdep.ddp.... cnt subs_allocated)
return 0;
Jurn_drive_fre (ree_subs(d))
ssubs_allocaters
return 1;
References
Table Pseudo code in stripped binary and correspond- ing refined pseudo code.
Avast Software. 2021. RetDec: A retargetable machine- code decompiler. https://retdec..om com/.
To intuitively demonstrate the effect of refined pseudo code, we provide concrete example in Table 6. In real world strip scenario, the pscudo code decompiled from assembly code often lacks descriptive function and variable names and in- stead uses placeholders such "sub_E6D78 "dword_162354"" To narrow the gap between pseudo code and natural language, utilized the fine-tuned CodeT5 to recover meaningful names and strings, such as "burn [burn_drive________ee_eee_eeeeee drivee_rree_e trre__sub"", "subs allocated"" which provide additional seman- tic information, even though the recovered strings may not be entirely accurate.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint 1607
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan, Association for Computational Linguis- tics.
YunSeok Choi, Jin Bak, Cheol Na, and Jee- Hyong Lee. 2021. Learning sequential and structural information source code summarization. In Find- ings of the Association for Computational Linguis- rics; ACL-IJCNLP 2021, pages 2842-2851, Online. Association for Computational Linguisties.
6 Conclusion
In this paper, we propose the CP-BCS framework, a novel approach that makes use of the control flow graph and pseudo code guidance. We manually construct the corresponding dataset that takes into account real-world scenarios. Finally, extensive experiments, ablation studies, and human evalua- tions demonstrate the effectiveness of CP-BCS. In practical applications, CP-BCS can significantly aid reverse engineers and security analysts in effi- ciently comprehending assembly code. We hope that our work can serve as a baseline while further prompting the development of this field.
Steven H. H. Ding, Benjamin C. M. Fung, and Philippe Charland. 2019, Asm2vec: Boosting static repre- sentation robustness for binary clone search against code obfuscation and compiler optimization. In 2019 IEEE Symposium on Security and Privacy (SP). pages 472-489
Han Gao, Shaoyin Cheng. Yinxing Xue, and Weiming Zhang. 2021. A lightweight framework for function name reassignment based on large- scale stripped bi- naries. In Proceedings of the 30th ACM SIGSOFT International Symposium Software Testing and Analysis, ISSTA 2021, page 607-619, New York. NY, USA. Association for Computing Machinery.

---

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang. Ming Zhou, and Jian Yin. 2022a. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 7212-7225, Dublin, Ireland. As- sociation for Computational Linguistics.
Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020. Improved code summariza- tion via graph neural network. In Proceedings of the 28th International Conference on Program Com- prehension, ICPC '20, page 184-195, New York, NY, USA. Association for Computing Machinery.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries, In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022b. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 7212-7225, Dublin, Ireland. As- sociation for Computational Linguistics.
Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-augmente generation for code summarization via hybrid (gnn). In Inter- national Conference on Learning Representations.
Juncai Guo, Jin Liu, Yao Wan, Li Li, and Pingyi Zhou.
2022c. Modeling hierarchical syntax structure with triplet position for source code summarization. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( Volume I: Long Papers), pages 360, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Grégoire Menguy, Sébastien Bardin, Richard Bonichon, and Cauim de Souza Lima. 2021. Search-basee local black- box deobfuscation: Understand. improve and mitigate. In Proceedings of the 2021 ACM SIGSAC Conference Computer and Communications Se- curity, CCS '21, page 2513-2525, New York, NY, USA. Association for Computing Machinery.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR).
OpenAL. 2022. Introducing chatgpt.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Hex-Rays 2021. 2021. IDA Pro. [Computer software].
Xing Hu, Ge Li, Xin Xia. David Lo, and Zhi Jin. 2018a. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, ICPC '1. page 200-210. New York, NY. USA. As- sociation for Computing Machinery.
J. Patrick-Evans, M. Dannehl, and J. Kinder. 2023. Xft: Naming functions in binaries with extreme multi- label learning. In 2023 2023 IEEE Symposium on Security and Privacy (SP) (SP). pages 2375-2390. Los Alamitos, CA, USA. IEEE Computer Society.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018b. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, ICPC 18, page 200-210, New York, NY, USA. As- sociation for Computing Machinery.
Moritz Schloegel, Tim Blazytko. Moritz Contag. Cor- nelius Aschermann, Julius Basler, Thorsten Holz, and Ali Abbasi. 2022. Loki: Hardening code obfuscation against automated attacks. In 31st USENIX Security Symposium (USENIX Security 22), pages 3055-3073, Boston, MA. USENIX Association.
Roshni Iyer, Yizhou Sun, Wei Wang. and Justin Gottschlich. 2020. Software language comprehen- sion using a program rerivee semantics graph. In NeurlPS 2020 Workshop Computer-Assisted Pro- gramming.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. 2022. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communica- tions Security, CCS '22, page 1631-1645, New York, NY, USA. Association for Computing Machinery.
Ensheng Shi, Yanlin Wang. Lun Du Hongyu Zhang. Shi Han, Dongmei Zhang, and Hongbin Sun. 2021. CAST: Enhancing code summarization with hierar- chical splitting and reconstruction of abstract syntax trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4053-4062, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Philipp Kochn. 2004. Pharaoh: a beam search de- coder for phsese-based statistical machine transla- tion models. In Machine Translation: From Real Users to Research: 6th Conference of the Associa- tion for Machine Translation in the Americas, AMTA 2004, Washington, DC, USA, September 28-0u 28-October 2, 2004, Proceedings 6, pages 115-124. Springer.

---

Jikyoeng Son, Joonghyuk Hahn, HaeonTae Seo, and Yo-Sub Han. 2022. Boosting code summarization by embedding code structures. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5966-5977, Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
A LLMs Assembly Code
Considering the emergence of large language mod- els (LLMs), such as ChatGPT (OpenAI, 2022), we made an initial attempt to explore their potential in understanding assembly code, Through numerous attempts, we discover that LLMs generally pos- sess only a rudimentary understanding of assembly code, such as memory operations and conditional jumps, as shown in Figure without any higher- level abstract semantic comprehension.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
nuv zon, [odil] teat can, CAKI ctpp. zep:
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Im- proving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE '18, page 397-407, New York, NY, USA. Association for Com- puting Machinery.
Auserbly
Goet
call
loa, wwwrewvel 7662222
[xaa]
[raxedth);
[caweddis))
CHeIGPT
[abx] naw zth. [sax]
call oub. v
[sbx]
call wub FTEerer0111TTe..
qeord pez [ebe). nov eax, b;
pop zbx: pop strp: patn
x01- sesemblly code that merrory operation and function calls. induding condition jumpa and netting registie
bofore rouning intoger value.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922.
Figure 6: Inputting assembly code into ChatGPT.
B Preliminary Survey
Yue Wang. Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empiricul Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
We conduct a preliminary investigation that aims to explore the types of binaries that reverse engi- neers encounter in their daily work, the binaries that have impeded their process, and the specific components that they are most concerned with dur- ing the reverse engineering process. We conduct a survey with 15 reverse engineers from academia and industry and analyzed the collected data using descriptive statistics and content analysis. Our find- ings indicate that reverse engineers face diverse range of binary programs, including both open- source and proprictary software, and encounter var- ious challenges that affect their productivity and effectiveness. The most common types of binaries reported by participants were operating system util- ities, drivers, and libraries. Regarding the specific components that reverse engineers are most con- cerned with during the reverse engineering process. our survey revealed that system-level functions, well networking and cryptography- com- ponents, are the most frequently cited ones.
Hongqiu Wu, Hai Zhao, and Min Zhang. 2021. Code summarization with structure-induced transformer. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1078-1090, Online. Association for Computational Linguisties.
Jia Yang, Cai Fu, Yiao-Yang Liu, Heng Yin, and Pan Zhou. 2022. Codee: tensor embedding scheme for binary code search. IEEE Transactions on Software Engineering, 48(7):2224-2244,
Tong Ye, Lingfei Wu, Tengfei Ma. Xuhong Zhang. Yangkai Du, Peiyu Liu, Wenhai Wang, and Shouling Ji. 2023. Tram: A woken-level retrieval- -uumented mechanism for source code summarization. arXiv preprint arXiv 1074.
Renyu Zhu, Lei Yuan, Xiang Li. Ming Gao, and Wenyuan Cai. 2022. neural network architecture for program understanding inspired by human behav- iors. In Proceedings of the 60th Annual Meeting of Computational Linguistics (Vol- une 1: Long Papers), pages 5142-5153.
Binary Projects
the Association for for
Table 7 displays a list of 51 binary projects and their corresponding versions.

---

Binary Projects
Version
Binary Projects
Version
Datasets (Arch: X86)
01
02
03
a2ps
4.14
binutils
2.30
Train
12,937
12,338
11,249
bool
0.2.2
ced2cue
0.5
Validation
1,617
1,542
1,406
cflow
1.5
coreutils
8.29
Test
1,617
1,542
1,406
epio
2.12
oppi
1.18
Assembly Code: Avg.
234.00
244.10
346.74
dap
3.10
datamash
1.3
BI-CPG: Avg. nodes
39,46
41.38
51,88
direvent
5.1
enscript
1.6.6
BI-CFG: Avg. edges
63.94
70.37
94.92
findutils
4.6.0
gawk
4.2.1
Pseudo Code: Avg. lokens
203.30
222.62
332.74
geal
4.1
gdhm
1.15
Summary: Avg. tokens
9.66
9.67
9.59
glpk
4.65
gmp
6.1.2
gnudos
1.11.4
grep
3.1
Table 8: Dataset statisties for X86 architecture with (O1,
gsasl
1880
2.5
02, and 03) optimization levels.
gss
1.0.3
gzip
1.9
hello
2.10
inctutils
1.9.4
libiconv
1.15
libidn2
2.0.5
Datasets (Arch: ARM)
01
03
libmicrohttpd
0.9.59
libosip2
5.0.0
Train
7,453
6,839
5,963
libtasnl
4.13
libtoul
2.4.6
Validation
932
855
745
libumistring
0.9.10
lightning
2.1.2
Test
932
8.54
745
macchanger
nettle
3.4
Assembly Code: Avg. tokens
276.87
279.26
390.37
patch
2.7.6
plotutils
2.6
BI-CFG: Avg. nodes
37.15
40.49
51,76
readline
7.0
recutils
1.7
BI-CFG: Avg. edges
58.96
67.42
90.34
sed
4.5
sharutils
4.15.2
Pscudo Code: Avg. tokens
241.36
269.40
387.84
spell
1.1
tar
1.30
Summary: Avg.
10.11
10.27
10.18
texinof
6.5
time
1.9
units
vmlinux
4.1.52
Table 9; Dataset statistics for ARM architecture with
wdiff
1.2.2
which
2.21
(O1, O2, and O3) optimization levels.
xorriso
1.4.8
Table The 51 binary projects and versions.
E Human Evaluation
For our human evaluation, we invited PhD stu- dents and reverse engineers as volunteers. All of our volunteers have at least 1-3 years of experi- ence in software enginecring and reverse engineer- ing. We randomly selected 200 examples from the dataset for volunteers to evaluate. The volunteers required answer the following questions.
D
Dataset Statistics and Explanations
Table 8 displays the statistics of our dataset on the X86 architecture for the three optimization levels. Table 9 displays the statisties of our dataset the ARM architecture for the three optimization levels.
How similar are the generated summary and ground-truth?
Similarity:
Currently, the dataset we've constructed is around the seale of 14k, and each sample has different variants (across three computer architec- tures and three optimization options). leading to total dataset size exceeding 100k. Compared to the source code summarization tasks where data collection is easier, the widely-used Java (Hu et al., 2018b) and Python (Wan et al., 2018) datasets have sizes of 70k and 80k, respectively. Although our dataset for a single architecture and single optimiza- tion option might appear smaller in comparison, there isn't a considerable difference in the order of magnitude. Notably, collected binary projects are diverse, encompassing domains such operat- ing systems, databases, and networking. Addition- ally, it's important to highlight that the assembly our dataset necessitates manual compilation process that is both rigorous and time-intensive.
Fluency: this generated summary syntacti- cally correct and fluent?
Time-Cost: The time and effort required understand assembly functions.
For Similarity and Fluency metric, the rating scale is from 1 to where higher score means better quality. For Time-Cost metric, we divide assembly code samples into five groups, each corresponding to one of the following scenarios: "Assembly Code Only", "Pseudo Code (CodeT5+)", "None", "Func- tion Name". and "CP-BCS", shown in the Figure 5. There are duplicates in the assembly code samples between any of the groups. We calculate the average time required by each volunteer to com- prehend each group of assembly code samples. To ensure faimess,, we attempt to maintain the same

---

02
Arch: ARM
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
Assembly
26.24
23.09
13.84
27.55
24.44
14.85
24.14
21.18
12.14
Pseudo (CodeT3)
24.66
24.25
13.44
25.22
25.20
13.00
23.97
22.90
13.33
CP-BCS w/o pseudo
28.24
25.64
15.56
29.01
26.19
14,89
25.45
22.48
13.52
CP-BCS w/o BI-CFG
28.82
26.48
16.30
28.67
26.56
15.56
24.76
21.86
12.59
CP-BCS w/o Refine
29.13
27.10
15.58
29.84
27.77
16.30
25.59
23.38
13.48
CP-BCS
29.75
27.84
16.81
29.56
27.67
15.98
26.66
24.26
14.03
Table 10: Baselines and ablation study results the dataset for ARM architecture.
01
02
03
Arch: X86
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
Assembly
21.69
16.98
9.64
18.24
12.72
6.64
21.52
17.48
9.27
Pseudo (SodeTT)
23.83
20.93
10.97
22.11
CP-BCS w/o pseudo
21.30
12.55
CP-BCS w/o BI-CRG
24.61
12.27
24.44
21.83
12.32
CP-BCS w/o Refine
25.53
23.64
13.31
CP-BCS
26.57
25.04
25.74
23.74
13.34
25.04
13.24
Table 11: Baselines and ablation study results on the dataset for X86 architecture.
01
02
03
Arch: X64
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
BLEU
ROUGE-L
METEOR
Assembly
22.88
18.82
11.09
21.52
16.96
8.77
22.53
19.03
11.01
Pscudo (CodeT5)
22.89
22.04
11.89
22.37
21.18
10.90
20.95
19.76
10.08
CP-BCS w/ pscudo
24.50
21.54
12.35
23.76
20.24
10.58
23.83
21.20
12.19
CP-BCS w/o BI-CFG
24.37
21.75
12.53
24.40
20.87
1 11.06
24.36
22.23
12.45
CP-BCS w/o Refine
25.61
23.20
13.12
24.96
21.98
11.89
24.31
21.78
12.52
CP-BCS
26.86
26.62
14.59
25.50
23.64
12.70
25.14
23.92
13.30
Table 12: Baselines and ablation study results on the dataset for X64 architecture.
number and length of assembly code instructions across all groups of samples much possible.
the Table 13, demonstrate that CP-BCS on new test set maintained similar performance, underscoring its scalability.
ARCH:X86; OPT:01
BLEU
ROUGL-L
METEOR
CP-BCS
13.50
CP-BCS on new test set
25.69
24.98
13.38
Table 13: Scalability Evaluations.
F Detailed Experimental Results
Table 10, Table and Table 12 show the experi- ment results of CP-BCS for three different archi- tectures and three different optimization levels.
To further demonstrate the scalability of CP- BCS, we conducted evaluations on approximately 200 newly compiled binary functions on X86 ar- chitecture and Ol optimization level (referred to as CP-BCS on new test ser). The results, presented in


Reviews:
[{"review_id": "1pyVQxR3KV", "reviewer": "Reviewer_hYNs", "summary": "", "questions": "1. The title of the paper is binary code summarazation, but it is actually assembly code summarazation. Is that appropriate?\n2. In a network structure, why is it pseudo code cross-attention, then assembly code cross-attention, and finally BI-CFG cross-attention?\n3. What is the result of concatenating assembly code and pseudo code directly?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the detailed and constructive comments. In the following, we will clarify the concerns point by point.**\n\n\n*Q1: The title of the paper is binary code summarization, but it is actually assembly code summarization. Is that appropriate?*\n\nA1: Thanks for pointing it out. There are primarily two reasons for this matter. Firstly, our research focuses on binary closed-source programs. The initial step in analyzing such programs usually involves translating them into assembly code. Secondly, within the context of practical reverse engineering, both the academic and industrial communities frequently employ the term \"binary code\" to denote related research areas, like the domain of binary code similarity[1][2]. In actual binary code similarity tasks, the operation target is also assembly code. Due to these factors, we chose to utilize the phrasing \"binary code summarization\" in the title of our work.\n\n[1] Jtrans: Jump-aware transformer for binary code similarity detection, ISSTA 2022\n\n[2] Order matters: Semantic-aware neural networks for binary code similarity detection, AAAI 2020\n\n*Q2: In a network structure, why is it pseudo code cross-attention, then assembly code cross-attention, and finally BI-CFG cross-attention?*\n\nA2: Thanks for the comment. Our choice of this particular order is driven by its slightly superior performance. However, in practical scenarios,  different cross-attention orders have minimal impact on the final performance. We take the dataset of the X64 architecture and O1 optimization level for example to provide concrete results:\n| Three Cross-attention Orders | BLEU | ROUGE_L | METEOR |\n| --- | --- | --- | --- |\n| assembly code→BI-CFG→pseudo code | 26.71 | 26.37 | 14.45 |\n| assembly code→pseudo code→BI-CFG | 26.50 | 26.40 | 14.39 |\n| BI-CFG→assembly code→pseudo code | 26.45 | 25.95 | 14.31 |\n| BI-CFG→pseudo code→assembly code | 26.47 | 26.08 | 14.49 |\n| **pseudo code→assembly code→BI-CFG** *(in our paper)* | **26.86** | **26.62** | 14.59 |\n| pseudo code→BI-CFG→assembly code | 26.86 | 26.45 | **14.67** |\n||\n\n\nBased on the experimental results, different orders only have a slight impact on the final performance (the BLEU score did not fluctuate by more than 0.5 points). We will incorporate the experimental results of different cross-attention orders into the Experiments Section of the paper to enhance the comprehensiveness and soundness of our results.\n\n*Q3. What is the result of concatenating assembly code and pseudo code directly?*\n\nA3: Thanks for the comment. Directly concatenating assembly code and pseudo code requires extending the maximum length of the encoder. We conducted experiments, and the results are as follows:\n| Arch: X64, OPT: O1 | BLEU | ROUGE_L | METEOR |\n| --- | --- | --- | --- |\n| concat(assem+pseudo) | 24.49 | 21.71 | 12.68 |\n| concat(assem+pseudo)+BI-CFG | 25.83 | 24.33 | 13.55 |\n| CP-BCS | **26.86** | **26.62** | **14.59** |\n||\n\nBased on the experimental results, direct concatenation can degrade the model's final performance to some extent, as shown in the last two rows of the table. Therefore, assigning a separate encoder for assembly code and pseudo code is a better choice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The first one proposes the binary code summarization task.\n2. The authors construct a parallel dataset containing different system architectures.\n3. CP-BCS method is obviously superior to baseline models and can improve the efficiency of reverse engineering.\n", "reasons_to_reject": "N/A", "questions_to_authors": "1. The title of the paper is binary code summarazation, but it is actually assembly code summarazation. Is that appropriate?\n2. In a network structure, why is it pseudo code cross-attention, then assembly code cross-attention, and finally BI-CFG cross-attention?\n3. What is the result of concatenating assembly code and pseudo code directly?", "paper_topic_and_main_contributions": "This paper focuses on assembly code summarization generation task. The authors first construct an assembly code - summary paired dataset containing three computer systems with three optimization levels. Then they present a control flow graph and pseudo code guided  binary code summarization framework called CP-BCS. The empirical results show that CP-BCS has good performance and can greatly improve the efficiency of reverse engineering.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yM8oFAgeJ4", "reviewer": "Reviewer_9guV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the detailed and constructive comments. In the following, we will clarify the concerns point by point.**\n\n*Q1: The dataset constructed by this paper is relatively small. It would be better to build a large dataset by some semi-supervised methods.*\n\nA1: Thanks for pointing this out. Currently, the dataset we've constructed is around the scale of 14k, and each sample has 9 different variants (across three computer architectures and three optimization options), leading to a total dataset size exceeding 100k. Compared to the source code summarization tasks where data collection is easier, the widely-used Java[1] and Python[2] datasets have sizes of 70k and 80k, respectively. Although our dataset for a single architecture and single optimization option might appear smaller in comparison, there isn't a considerable difference in the order of magnitude. Moreover, to ensure our dataset is compatible across the three computer architectures and three compilation optimization options, we have to perform manual compilation and choose the appropriate compilation configurations. At this current stage, this remains essential.  At the same time, we're continuously collecting and building a larger dataset. In line with your recommendation, we're delving into semi-supervised strategies, such as presetting configuration parameters. Once completed, we'll make it open-source on the corresponding GitHub repository to facilitate further research.\n\n[1] Deep code comment generation, ICPC 2018.\n\n[2] Improving automatic source code summarization via deep reinforcement learning, ASE 2018.\n\n*Q2: Other SOTA code summarization methods, like using pre-trained models, should be compared in the experiments.*\n\nA2：Thanks for the comment. We employed code-related pre-trained models (CodeT5 [1], CodeT5+ [2], and UniXcoder [3]) for fine-tuning to compare with CP-BCS. The experimental results (on X64 architecture and O1 optimization level), as shown in the table below, indicate that pre-trained models perform well, with a notably better performance on pseudo code than assembly code. Intuitively, this makes sense as pseudo code is more akin to high-level programming languages compared to assembly code. Overall, our proposed CP-BCS demonstrates a superior performance compared to direct fine-tuning of pre-trained models.\n\n| Arch: X64, OPT: O1 | BLEU | ROUGE_L | METEOR |\n| --- | --- | --- | --- |\n| assembly code (CodeT5-base) | 21.87 | 20.78 | 11.05 |\n| pseudo code (CodeT5-base) | 22.89 | 22.04 | 11.89 |\n| assembly code (CodeT5+, 220M) | 23.07 | 21.02 | 11.58 |\n| pseudo code (CodeT5+, 220M) | 24.14 | 23.83 | 12.48 |\n| assembly code (Unixcoder) | 22.01 | 18.48 | 10.48 |\n| pseudo code (Unixcoder) | 23.17 | 22.65 | 12.35 |\n| CP-BCS | **26.86** | **26.62** | **14.59** |\n||\n\nWe will include the experimental outcomes in the Experiments Section of the paper to enhance the comprehensiveness and soundness of our results.\n\n[1] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation, EMNLP 2021\n\n[2] CodeT5+: Open Code Large Language Models for Code Understanding and Generation, arXiv 2023\n\n[3] UniXcoder: Unified Cross-Modal Pre-training for Code Representation, ACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. This paper constructs and releases a comprehensive assembly code summarization dataset. It will facilitate the development of this direction.\n\n2. They design a multi-source encoder framework to make full use of the control flow graph and pseudo code guidance in binary code summarization.\n", "reasons_to_reject": "\n1. The dataset constructed by this paper is relatively small. It would be better to build a large dataset by some semi-supervised methods.\n\n2. Other SOTA code summarization methods, like using pre-trained models, should be compared in the experiments.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThis paper proposes a control flow graph and pseudo code guided binary code summarization method, which extracts the execution behavior and semantics of assembly code. The proposed model utilizes three different encoders to model assembly instruction, BI-CFG and pseudo code, separately, and employs a decoder to generate the final summarization. They also construct and release a binary code summarization data including assembly code and summary pairs. Experiments with automatic metrics and human evaluation demonstrate the effectiveness of the proposed method.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SZJQKBrGK0", "reviewer": "Reviewer_bpsr", "summary": "", "questions": "- While different components of the model are described in the Methodology section, there should also be some motivation as to why an autoencoder based architecture is used e.g., why not consider an encoder only architecture with an classier such as MLP component?  \n- The size of the datasets are in the thousands, which seem to be quite small. As such, I wonder what the scalability of the proposed approach is. Can the authors provide a section regarding the time complexity of their model? ", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the detailed and constructive comments. In the following, we will clarify the concerns point by point.**\n\n*Q1: There is lack of motivation as to why the problem of investigation is important. ······ Alternatively, what are the different applications that can be benefited with solving this problem.*\n\nA1: Thanks for your suggestion. In today's reverse engineering domain, analyzing assembly code is a daunting task for security experts and reverse engineers, as they also grapple with decrypting unknown protocols, dissecting embedded firmware behaviors, and identifying hidden vulnerabilities in software. Assembly code, being a low-level representation of machine instructions, typically lacks the readability and structure of high-level programming languages. This means that security analysts must decipher the meanings of these instructions to understand the behavior of the program. Therefore, there is a pressing need for tools that can generate clear and concise summaries of binary programs. Our CP-BCS has the ability to automatically generate human-readable natural language summaries for binary functions. This feature significantly streamlines the comprehension process for both reverse engineers and security analysts alike. Human evaluations have shown that, when using CP-BCS-generated summaries, the productivity of security analysts can increase by a significant factor of 9.7, as detailed in Section 5.4. Such an advancement holds promise for boosting efficiency across various applications, including bug detection in binary programs, binary code optimization, and root-cause analysis.\n\n*Q2: In the Methodology section, it would be helpful if the authors could introduce a Problem Formulation section to formally describe the problem to be solved and any relevant assumptions and constraints.*\n\nA2: Thanks for pointing out this issue. We will incorporate the following Problem Formulation into the paper for greater clarity. \n\n**Problem Formulation:** \n\nGiven a binary function with $l_c$ assembly code tokens $T_c = (c_1, c_2, ···,c_{l_c})$, and its BI-CFG with $l_n$ nodes $T_n = (n_1, n_2, ···,n_{l_n})$ , and its corresponding pseudo code with $l_p$ tokens $T_p = (p_1, p_2, ···,p_{l_p})$, CP-BCS predicts the next summary tokens based on the existing tokens $T_s = (
, s_1, s_2, ···, s_{m-1},···)$，where $
$ is a special starting tag for summary input.\n\nThe objective of our research is to generate a concise natural language summary for the binary function. Like the binary function shown in Figure 1 (*\"mov rax, [rdi] test rax, ······, retn\"*)，our target is to generate the natural language summary: *\"free all resources associated with context_handle.\"*\n\n*Q3: While different components of the model are described in the Methodology section, there should also be some motivation as to why an autoencoder based architecture is used e.g., why not consider an encoder only architecture with an classier such as MLP component?*\n\nA3: Thank you for the comment about the model architecture. As discussed in the introduction section (line 66), considering that the binary code summarization task involves generating a natural language text summary, which is a generative task essentially, an encoder-only architecture is insufficient to achieve the task’s objectives. Thus, we have chosen an encoder-decoder architecture. The encoder component aims to comprehensively extract behavior and logic semantics across three different dimensions (assembly instruction, BI-CFG, and pseudo code). The decoder component is designed to generate corresponding summary tokens in an autoregressive manner.\n\n*Q4: The size of the datasets are in the thousands, which seem to be quite small. As such, I wonder what the scalability of the proposed approach is. Can the authors provide a section regarding the time complexity of their model?*\n\nA4: Thanks for highlighting these points. At present, the dataset we've constructed is around the scale of 14k, and each sample has 9 different variants (across three computer architectures and three optimization options), leading to a total dataset size exceeding 100k. Compared to the source code summarization tasks where data collection is easier, the widely-used Java[1] and Python[2] datasets have sizes of 70k and 80k, respectively. Although our dataset for a single architecture and single optimization option might appear smaller in comparison, there isn't a considerable difference in the order of magnitude. Notably, our collected binary projects are diverse, encompassing domains such as operating systems, databases, and networking. Additionally, it's important to highlight that the assembly of our dataset necessitates manual compilation—a process that is both rigorous and time-intensive. Concurrently, we're continuously building a larger dataset and will share it on the corresponding GitHub  repository soon.\n\n[1] Deep code comment generation, ICPC 2018.\n\n[2] Improving automatic source code summarization via deep reinforcement learning, ASE 2018.\n\n**Scalability:**\n\nTo further demonstrate the scalability of our proposed approach, we conducted evaluations on approximately 200 newly compiled binary functions on X86 architecture and O1 optimization level (referred to as *CP-BCS on new test set*). The results, presented in the table below, demonstrate that CP-BCS on new test set maintained similar performance, highlighting its scalability.\n\n| Arch: X86, OPT: O1 | BLEU | ROUGE-L | METEOR |\n| --- | --- | --- | --- |\n| CP-BCS | 26.57 | 25.04 | 13.50 |\n| *CP-BCS on new test set* | 25.69 | 24.98 | 13.38 |\n||\n\n**Time complexity:**\n\nThe time complexity of CP-BCS model depends on three encoders (assembly instruction encoder, BI-CFG encoder, and pseudo code encoder) and one summary decoder. \n\n+ For the assembly instruction encoder and pseudo code encoder, both employ the transformer encoder design. Assuming the length of the assembly code length and pseudo code length are both $n$ and the feature dimension is $d$,  their time complexities stand at $O(n^2d)$. \n+ The BI-CFG encoder, based on a graph attention network, has its complexity determined by the number of nodes  $v$ and edges $e$ in each binary function, with a node feature dimension of  $d$. Its complexity is  $O(vd^2+ed)$. \n+ The summary decoder’s predominant time complexity arises from its three cross-attention modules, making its time complexity $O(2*n^2d+v^2d)$.\n\nIn our empirical evaluations, generating summaries for 1,000 test samples took around 33 seconds, which we consider to be highly efficient and capable of meeting practical requirements.\n\n*Q5: Missing References*\n\nA5: Thanks for pointing out this issue. We promise to include this reference “Software Language Comprehension using a Program-Derived Semantics Graph (NeurIPS-CAP 2020)” in the Related Works section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The Introduction and Related Works sections present some summary of limitations of existing works, as well as contributions of CP-BCS. \n- The Methodology portions partitions the various components of the architecture well, with components of the: assembly instruction encoder, BI-CFG Encoder, Pseudo Code Encoder, and Summary Decoder. \n- Dataset statistics are provided for broad range of datasets and computer architecture", "reasons_to_reject": "- There is lack of motivation as to why the problem of investigation is important. For example, the authors may consider answering questions like what the benefits are with generating complete summaries for binary functions from an efficiency or accuracy point of view.  Alternatively, what are the different applications that can be benefited with solving this problem. \n- In the Methodology section, it would be helpful if the authors could introduce a Problem Formulation section to formally describe the problem to be solved and any relevant assumptions and constraints. ", "questions_to_authors": "- While different components of the model are described in the Methodology section, there should also be some motivation as to why an autoencoder based architecture is used e.g., why not consider an encoder only architecture with an classier such as MLP component?  \n- The size of the datasets are in the thousands, which seem to be quite small. As such, I wonder what the scalability of the proposed approach is. Can the authors provide a section regarding the time complexity of their model? ", "paper_topic_and_main_contributions": "This work focuses on generating complete summaries for binary functions, particularly stripped binary. To fully exploit the semantics of assembly code, authors present a control flow graph and pseudo code guided binary code summarization model, CP-BCS. CP-BCS uses a bidirectional instruction-level control flow graph and pseudo code with expert knowledge to learn binary function execution behavior and logic semantics. The authors evaluate CP-BCS on three different binary optimization levels (O1, O2, and O3) for three different computer architectures (X86, X64, 025 and ARM).", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In the Related Works section, there is a recent work on understanding and capturing semantic meaning from code through a program-derived semantics graph. The authors should reference this work in this section: \n- (NeurIPS-CAP 2020) Software Language Comprehension using a Program-Derived Semantics Graph, https://arxiv.org/abs/2004.00768\n"}}]

Author rebuttal:

Meta-review: This paper introduces a novel framework, CP-BCS (Control Flow Graph and Pseudo Code Guided Binary Code Summarization), for the task of assembly code summarization. The authors build a dataset containing assembly code-summary pairs across three computer systems with different optimization levels. CP-BCS uses a bidirectional instruction-level control flow graph and pseudo code to learn the execution behavior and logic semantics of binary functions. The method is evaluated on three different binary optimization levels for three different computer architectures, showing considerable improvement in the efficiency of reverse engineering.

Main Contributions:

The paper introduces the task of binary code summarization.
The authors construct a parallel dataset with assembly code-summary pairs across different system architectures.
A new framework, CP-BCS, is proposed, which uses a multi-source encoder strategy to leverage control flow graphs and pseudo code for binary code summarization.
The authors provide comprehensive dataset statistics across a broad range of datasets and computer architectures.

Reasons for Acceptance:

The introduction of the binary code summarization task fills a gap in current research.
The construction and release of a comprehensive assembly code summarization dataset will facilitate further development in this field.
The CP-BCS method shows superior performance to baseline models, improving the efficiency of reverse engineering.
The methodology is well-described and partitioned into various components, including the assembly instruction encoder, BI-CFG Encoder, Pseudo Code Encoder, and Summary Decoder.
The paper provides a thorough evaluation of CP-BCS on different binary optimization levels and computer architectures.
Decision: Accept-Main