Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases
Authors: Zian Su
Xiangzhe Xu
Ziyang Huang
Kaiyuan Zhang
Xiangyu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Code Language Models
Human-Understandable Binary Recovery
Cross-Modal Prompting
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Human-Oriented Binary Reverse Engineering (HOBRE) lies at the intersection of binary and source code, aiming to lift binary code to human-readable content relevant to source code, thereby bridging the binary-source semantic gap. Recent advancements in uni-modal code model pre-training, particularly in generative Source Code Foundation Models (SCFMs) and binary understanding models, have laid the groundwork for transfer learning applicable to HOBRE. However, existing approaches for HOBRE rely heavily on uni-modal models like SCFMs for supervised fine-tuning or general LLMs for prompting, resulting in sub-optimal performance. Inspired by recent progress in large multi-modal models, we propose that it is possible to harness the strengths of uni-modal code models from both sides to bridge the semantic gap effectively. In this paper, we introduce a novel probe-and-recover framework that incorporates a binary-source encoder-decoder model and black-box LLMs for binary analysis. Our approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments as context. This additional context enables black-box LLMs to enhance recovery accuracy. We demonstrate significant improvements in zero-shot binary summarization and binary function name recovery, with a 10.3% relative gain in CHRF and a 16.7% relative gain in a GPT4-based metric for summarization, as well as a 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results highlight the effectiveness of our approach in automating and improving binary code analysis.

Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases
Zian Sul Xiangzhe Xu'
Ziyang Huang² Kaiyuan Zhang' Xiangyu Zhang'
Purdue university Johns Hopkins University
2aa.a4...22ann4477eaadrrad edu, zhuang86@jhu. edu
xyzhang@cs purdue. edu
Abstract
Human-Oriented Binary Reverse Engineering (HOBRE) lies at the intersection of binary and source code, aiming to lift binary code to huaanlreadable content relevant to source code, thereby bridging the binary-source semantic gap. Recent advancements in uni-modal code model pre-training, particularly in generative Source Code Foundation Models (SCFMs) and binary understanding models, have laid the groundwork for transfer learning applicable HOBRE. However, existing approaches for HOBRE rely heavily on uni uni-modal models like SCFMs for supervised fine-tuning or general L.MSs for prompting, resulting in sub-optimal performance. Inspired by recent progress in large multi-modal models, we propose that it is possible to harness the strengths of uni-modal code models from both sides to bridge the semantic gap effectively. In this paper, we introduce a novel probe-and-recove framework that incorporates a binary-source encoder-decoder model and black-box LLMs for binary analysis. Our approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments context. This additional context enables black-box LLMs to enhance recovery accuracy. We demonstrate significant improvements in zero-shot binary summarization and binary function name recovery, with 10.3% relative gain in CHRF and 16.7% relative gain in a GPT4-based metric for summarization, well 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results highlight the effectiveness of our approach in automating and improving binary code analysis.
Introduction
In recent years, we see two trends of uni-modal code model pre-training. On one hand, there is
remarkable surge in the development of generative Source Code Foundation Models (SCFMs) 12 655
55 20 43). along with advancements in general Large Language Models (LLMs) 4 60 46). Driven
by a growing interest in automating software development, these powerful models are trained on billions of tokens from diverse codebases, covering a wide spectrum of programming languages 33
They possess the capability to complete, infill [9), and refine code [67] as well as generate code from natural language instructions [44). On the other hand, there is stream of research focusing binary understanding models [6] 58), which target learning nuanced code semantics with structures of low-level code, which is critical for software security. Both fields evolving through continuous pre-training on expansive datasets of uni-modal data, setting new benchmarks in both effectiveness and complexity.
Human-Oriented Binary Reverse Engineering (HOBRE), which involves automatically lifting bi- nary to human understandable contents []4. 69. typically source-code related, occupies unique
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hinary Function 2
Black-Box LLMs
1nt84 Fastcall sib, 4010CA. 1
OLL;
Recoverer
A
resalt -11
Probed Context 8
breaks
wid perCintt4 "4 (NLSLLL 7b, ...)
(ketes)))
C3 85 192
The polenial seurce fanction i liaplly relevast decorppiiee spuhtinn cirbbr
(L00)))((-11)))) (8-1 "(JNaor)((*+aa)
perform seays
=[i]:
ProRec
Cross-Modar Knowledge Prober
wild 2
The purpose of the lunction secms to be to update the memory locations pointed to by al and 22 brood on the Iteration count and the values stured those locations. likely performs sume of cumulative calculation or data manipulacion.
gidsums for Cf oi 1ee3
oldeuse daralels sumsf13)
Ornele Sourec Function
#2
g
cyTaa cindecideeeet 1dxtnce
Direct Prompting
unsigned long "y, long long length)
The puspose the lunction seems to manspulase the memory block by swapinn and updating values based en the index
1dxtne cumsurs T1
long
hort sub: 1 o un 14+ X
und unsil Tony
cumpans
cansus [i)
for 0: length: 1-4) E
WES) #L110
x[i]
Source Cude Summarisation
v113 72
purpose camulative e sum the elements in the input army w and store the results in the army while also upéining the values the w arroy.
calculate the
indevut
relevatt
sinsiler loopo
Figure 1: The ProRec Framework for oriented binary reverse engineering. The figure shows a simple example of lifting a cumsum function from binary to human readable summarization. The probed contexts synthesized by the cross- mosdll knowledge prober, while not identical to the oracle source code of the query binary, exhibit informativeness in terms of symbol names and correct loop structure. These contexts help the black-box LLMs successfully recover the high-level functionality of binary function in the summary that is consistent with the source code summary, moving beyond merely describing its low-level operations.
intersection between these two fields. Existing decompilers for binary reverse engineering are able to translate binary code to C-style code that is functionally equivalent to its original source code. However, significant semantic gap remains between the decompiled code and its original source. primarily due to the absence of meaningful symbolic information in binary code. Hence, human expertise is still indispensable in the reverse engincering process. HOBRE aims to bridge this se- mantic gap, which traditionally requires substantial human effort, by leveraging cross-modal deep learning models. Existing approaches either train task-specifer small expert models in a supervised manner (30, 1 69. (75), which lack generalizability as shown in later evaluations [56], or require extensive continual pre-training of uni-modal SCFMs (28) which is undesirable considering cost and the risk of forgetting previously acquired source code knowledge [2, 777. There are also attempts in directly prompting L.MSs for HOBRE, which, even though demonstrates better generalizability than small supervised models, also face challenges in understanding stripped decompiled code that lacks symbolic information [29].
Our insight is that this semantic gap between binary and source code is analogous to the gap between lowelevel pixels in images and high-level concepts in natural language, which can be bridged with sufficient understanding of both. Inspired by the achievements of molti-modal models that seamlessly integrate vision, audio, or other signals with language to facilitate reasoning 37 45 we hypothesize that HOBRE could similarly benefit from leveraging uni-modal models developed for both source code and binary code. Such integration would enhance our ability to bridge the semantic gap and enable more effective semantic lifting.
In this paper, we validate this idea by proposing a novel probe-and-recover framework ProRec that incorporates a binary- encoder-decoder model and black-box LLMs for HOBRE, featuring a compute-efficient mross alignment approach of binary function encoder and frozen SCFM for the binary-souree model. The workflow of ProRec is shown in Figure T The aligned binary- source model acts as a cross-modal-king shooledge-probble code fragments condition on binary input, denoted as probed contexts. The blxck-box LLM functions as recoverer that takes as input the binary function together with the probed contexts for tasks such as binary summarization. Intuitively, the conditional source code synthesis by the aligned bonary-source code model can be viewed as probing the base SCFM as a parametric knowledge base [52] with a binary function as query, given that the SCFM's weights remains unchanged before and after the
that can synthesize symbol-rich, diverse source

---

alignment. A black-box LLM analyzes and aggregates these knowledgable contexts with the binary function for recovery. This way, ProRec leverages both cross-modal aligned knowledge and strong reasoning ability of L.LMs and can outperform directly letting the L.LM to reason. ProRec is general and can be applied different hase architectures, continually evolve with base models.
We demonstrate the effectiveness of ProRec on two core tasks in reverse engineering [9 binary summarization and binary function name recovery. The former aims generate natural language descriptions for a binary function, and the later aims to recover the function name of decompiled function. We evaluate ProRec on diversified dataset compiled from GitHub repositories. demonstrating improvements of 3.1% (10.3% relative gain) in CHRF and 12% (16.7% relative gain) in GPT4-based metric that has high correlation with human judgement on the summarization task over zero-shot baseline. We conduct human study to show the effectiveness of the newly proposed GPT4-based metric. On name recovery tasks, ProRec significantly improves over zero-shot baseline by 6.7% and 7.4% for token-level precision and recall, respectively. For both tasks, ProRec also consistently show advantage over a retrieval- -uumentl baseline with strong cross-modal dense retriever.
2 ProRec: Reverse Binary by Probing Source Code Foundation Models
In this section, we first present the Proßec framework in 2.1 Next, we describe the neural architecture used for the cross-modal knowledge prober and recoverer in $2.2 The training for the prober in is detailed in $2.3 followed by the comphrehensitve explanation of the knowledge probing stage in 2.4
Formulation Given a binary file, we can leverage binary analysis tools to obtain each binary function Specifically, either be in its disassembled code form which denote Taans or its stripped decompiled code form, denoted as Idec- aasn and daec are semantically equivalent and similarly unreadable. The goal is to recover human readable information given Idec and Taam.
2.1 The Frobe-and-Recovee Framework
ProRec assumes binary understanding model parameterized by an open-source SCFM parame- terized by db, and a black-box LLMs by 6. As illustrated in Figure [ the binary model together with the SCFM form the cross-modal knowledge prober. The black-box L.M serves as a recoverer. The cross-modal prober can synthesize source code fragments given binary input. The recoverer takes in augmented context with binary code to analyze and perform final recovery.
Conceptually, the ProRec framework decomposes the probability to generate into three parts, the probability of set of k source code fragments S& {s1,... 8& being relevant input (()) the probability of LLM's relevance analysis of the source code fragments P(A|Sx,*)) x), and the probability of generating the recovery results conditioned on the analysis and source code fragments.
P(y|z)
Ps (vlA, Su~ P.-
Σ
(AIS,,)) P (Sular
(I)
The decomposition is similar to that of augmental- generation 36 68 where p(y|æ) = Eestop-((S-) P(yls, 2:) P(s)2:) given document pool S However, there are two major differences. First, the source code fragments S₁ are not retrieved from S*, instead, they are sampled from the conditional distribution of the prober Po.v Due to the alignment strategy (discussed in (2.3), source code fragments sampled from the prober's 's distribution have more flexibility than those retrivedd a fixed document pool in binary reverse engineering scenario, potentially less noisy. We empirically demonstrate the superiority of probing over retrieval for augmentation in $4
Second, we stress the internal analysis from the LLM denoted as Po (A]Se, the decomposition. The insight is that, even though high-level recovery requires additional domain information to hint black-box LLMs for further induction, that doesn't necessarily mean LLMs totally lacks of such
"Orr code and data are available at https: hhex- rays.com//t........ www.ecammmaa//e/eeeeee..ee.eeaae itaaaddaa////aa
3

---

stre_t n,...
Source Code Foundation Model
NLL Estimatiin by SCFM
3.5
Z
3.0
2.5
Projection Layer
Word Embedding Layer
2.0
void hsort(void
1.5
Zun
Pizze
Dataddings
1.0
Last Block
say rnl,, nad
Thoam
Structure-Awan Binary Function Encoder
Tea Indearacd)
Inarabe
0.0
Backoond
Estimation by Aligned Prober
Gárs
Fass
Figure 2: The prober architecture and compute- efficient alignment with limited trainable parame- ters.
Figure 3: Negatíve log-likelihoods of source functions estimated by base SCLM and those conditioned its binary counterpart esti- mated by the aligned prober.
knowledge (since proprietary LLMs can be significantly larger than open-source code language models in size and may have more training data, just different mixtures), it might be some long-tail knowledge that requires better prompting to exploit [35] (6). On the other hand, the analysis help LLMs to be less influenced by the noisy contexts. This is beneficial for both retrieved and probed contexts.
Note that, in Equation] 1 the final probability is marginalized over all possible S₁ and A. In practice, we take the most probable S₁ and keep the analysis with in the response before final result, without heavy sampling. We will discuss the sampling of each within S₁ in $2.4
2.2 Model Architecture and Instantiation
ProRec a general framework and bounded existing models and architectures. This section is our current implementation of the prober and recoverer that provide the best performance in our experiments.
Cross-Modal Knowledge Prober The core ProRec is the cross-modal prober, which is an encoder-decoder model aligned in the token embedding space of the SCFM, illustrated in Fig- ure 2
We would like to benefit from both pretrained binary function encoders that possesses binary domain knowledge and the strong SCFMs for generalizable probing. We choose the state-of- oatte-of-the-ar CODEART [58] as our structure-ane aware binary function encoder 9(.). CODEART is a BERT-like transformer enccoder that takes input disassembled binary function Fasn along with its depen- dency graph Gaap obtained by program analysis, and outputs the emheddings for all assembly code tokens and graph node tokens (each graph node token corresponds to instruction, e.g., mov rax, [xbp+var__] _4]] the assembly code). We choose the Code-Llama (55 family as our base SCFM
For the final prober architecture, we apply simple two-layer MLP project the node token embeddings asn with indices IDX all token embeddings, to source code token embeddings
space.
Zam CODEART (Iamn: Gaap) [N_IDX, Rlaxds Zpro = MLP(Z R xd,
(2)
where I,, denotes the number of node tokens, d, is the dimension of the binary encoder and d, is the dimension SCFM. The projected embeddings are fed into the SCLM as an additional prefix before regular subtoken embeddings for conditional generation.
*We also tried other SCFMs like DeepSeek-Coee Coder 20 StarCoder2 that Code-Llama performs best base SCFM for our prober.
43 in our preliminary study and find

---

We only use node token embeddings as binary features due to their significantly smaller quantity compared all token embeddings (approximately one eighth) since assembly functions tend be long. These embeddings also already capture some structural abstraction of binary code which is meaningful in HOBRE tasks.
Recoverer We leverage proprietary black-box LLMs (GPT3.5, Claude-3, and Gemini-Pro) as our recoverer, since they have strong reasoning ability and support long contexts. Specifically, the LLMs prompted with deec as zero-shot baseline. For retrieval-augmented baseline and ProRec, append the additional context to the original input and instruct LLMs to analyze relevance and then generate recovery. Detailed prompts can be found in Appendix D
2.3 Prober Training
The training of prober contains two stages; the pre-alignment of the binary encoder to source code encoder, and the binary Seooder-SCFM alignment. Both utilize data in the form of paired binary-source functions. The goal is to gradually align the binary encoder with the base SCFM with minimum knowledge loss.
Contrastive Assembly-Source Code Pre-Alignment Since CODEART is exclusively pre- trained on binary code corpus [58], we first align it with pre-trained source code encoder -ereeddpp-embedddn------ 1100 64 in the function-level embedding space as pre-alignment stage in order to facilitate the later encoder-decoder alignment. To achieve this, we add projection head for each encoder to project their [CLS] token embeddings to the same dimension denc, forming a standard dual-encoder [27]. This dual-encoder can encode (Taan, Gāep) into haan E Rd and Tarc into hare E dd We train the dual-encoder in CLIP-like symmetric contrastive fashion [54] Since the implementation is relatively standard, we refer readers to Appendix [A] for details.
The dual-encoder can function for a query binary function from the source function pool of the training set, based on the similarity measure sim Pasm: ( = cos (ham, htrr). It achieves 84% recall@ @ on the validation set with pool of 10k examples, demonstrating strong performance as a retriever. We utilize this dual-encoder to set up retrieval-anie larreval-augmeented HOBRE baseline to compare with ProRec in $4
dense retriever to score and retrieve the top-k source functions
Compute-Efttcient Cross-Modal Prober Alignment For encoder-decoder alignment, we freeze all the parameters within the SCFM because we intend explore the extreme end of probing knowledge from it. We freeze CODEART from the first stage except for the last layer which is transformer block for fast convergence and avoid too much change in the representation. The MLP is fully trainable. The objective of the alignment is to maximize
[Farc] II (o. proj,
P(Iare (aams Gdep)
The limited amount of trainable parameters results in efficient training. For memory efficiency, apply quantization (4bit or 8bit) [8) to the base SCFM during aligmment.
One evidence that the knowledge of the aligned prober is mainly from the SCFM pre-training instead of learned during alignment is shown in Figure 3 We sampled 500 (#azm were) pairs from the validation set and find that the negative log-likelihood - log (TTee)) for Tare provided by the base SCFM and log Po.u((Zes @aaem Gaep) for Øere conditioned on the aasn provided by the aligned prober are highly correlated, indicating that the prober's ability is consistent with the base SCFM. Another interesting observation is that, instruction-tuned SCFMs typically show higher losses during alignment than their original models, which also implies the significance of pre-trained knowledge of source code for cross-modal ability as instruction-tuning may cause forgetting.
2.4 Cross-Modal Knowledge Probing
For the probing process, i.e., sampling S₁ with the aligned P- Gaep), ) we want to cover a diverse yet relevant set of candidates. We leverage nucleus sampling (24 to first let the prober generate
5

---

relatively large set of source function signatures with high randonmess (top-p = 0.75). We use idea similar retrieval by training a binary-signature dual-encoder to rank generated signatures and filter out the noisy ones. Ultimately, we use the prober to further complete the remaining signatures with smaller randomness (top-p = 0.5). Since signature is short and important for HOBRE, our strategy achieves both better relevance compared to using a fixed small p for full function generation and better efficiency compared to sampling a large set functions with large p.
3 Experiment Setup
We evaluate ProRec on two binary reverse engineering tasks: summarizing function semantics from decompiled code (33.1). and recovering function names from decompiled code ( $.2 . In this section, we first introduce our dataset, and the setups of each task.
Dataset The training and evaluation of ProRec requires pair-wise data between binary function and its corresponding source code. To the best of our knowledge, there is no publicly available dataset that contains matched source code with the binary program. Therefore, we follow widely adapted practice in the reverse engineering domain [3 compile repositories from GitHub. After the compilation, we map the resulting binary programs with the corresponding source code functions leveraging the debug information in binary programs. In total, our data consists of 270k pairs of binary and source code functions. We split 260k data samples for training and 10k data samples for test. We use 5% of the training data as the validation dataset. To make the evaluation cost tractable, we randomly sample Ik samples from the test dataset. For details in data processing and quality assurance, please see Appendix B.
34 77.. using GHCC to automatically clone and
3.1 Binary Summarization
binary summarization tool takes as input a snippet of decompiled code, and outputs natural language descriptions. It facilitates two key processes in the reverse engincering practice [9]: understanding the purpose and domain of program (referred to as context relevance), and understanding the functionality of a program (functionality). Please see Appendix F for detailed example.
Setup We instruct an LLM to summarize decompiled code with three setups: (1) providing the model with only the decompiled code; (2) additionally providing the relevant source code snippets retrieved from the datastore consisting of all source functions in prober's 's training set by the cross- modal retriever; (3) additionally providing and the source code snippets generated by ProRec. The first two setups are considered bascline approaches for comparison. We further instruct each LLM to summarize the source code corresponding the test samples as reference summaries.
Metrics Given the reverse engineering nature of binary summarization, the automatic evaluation metrics should reflect context relevance and functionality of the summary, different from text summa- rization. For final results, report CHRF [53], which our meta-evaluation (described next) identified as the most aligned with human preferences among popular existing metrics such as BLEU [48). Additionally, introduce and report two Gased metrics for context relevance and functionality judgement respectively, following LLM as a Judge [78), which demonstrate strong correlation with human judgments. The GPT4-based metries range from 1 (worst) to 5 (best) based on corresponding criteria. Further details (e.g., prompts and rationale) about the GPT4-based metrics can be found in Appendix B.2 and Appendix D
User Study We conduct a user study to gather human judgments on the quality of binary summarization, which serves as the gold standard for this task. The study aims to (1) perform meta- evaluation of automatic metrics and (2) accurately assess the performance of different summarization approaches. Participants are asked to score summary based on decompiled code, corresponding source code, and the reference summary. The scoring is done on two intteria-context relevance and functionality-on scale from (worst) to 5 (best). The method used to generate each summary is not disclosed to the participants. For the meta-evaluation of automatic metrics, we calculate the
hattpa://githul httpp/////gihabb.cmm//ueeeee wtt.//ow.../////eeeeee com/huzecong/ghcc.
We obtained an IRB for the study (IRB-2024-799)

---

Table 1: Main results of binary summarization and binary function name recovery. "G4-F" and "G4-C" denote GPT4Evaluator for functionality and context relevance, respectively. Psyml M. M M denote token-level precision, recall, F-1 score as SymLM [30]. "cBLEU", "cRol." and "cMETEOR" stands for character-level BLEU, ROUGE-I, and METEOR scores.
Summarization
Function Name Recovery
CHRF
G4-F
G4-C
SyNLM
R
FSymLM
cBLEU
cRoL
cMETEOR
GPT-3.5-turbo GPT-3.5-turbo +retrieval
30.4
3.6
3.8
16.3
20.9
17.2
11.9
45.1
38.1
31.7
3.7
15.5
21.3
17.0
10.8
45.5
38.3
GPT-3.5-turbo
+ProRec
33.5
4.2
4.0
22.2
28.3
23.5
14.4
47.6
41.1
Gemini Pro Gemini Pro Gemini- Pro
27.1
3.7
3.5
25.3
28.7
25.3
16.8
48.1
39.5
+retrieval +ProRec
26.4
3,4
3,2
22.7
23.3
21.6
16.2
45.8
36.5
27.6
3.8
3.6
32,6
30.5
29.9
22,4
50.9
40.9
Claude-3 Claude-3 +retrieaal
33.5
3.7
3.9
16.5
22.3
17.9
13.2
40.7
33.9
33.9
3.8
3.9
19.9
23.9
20.5
14.0
43.3
36,0
Claude-3
+ProRec
4.0
25.9
29.4
17.8
40.1
Spearman correlation between human judgments and automatic metric scores. For details, refer readers to Appendix
3.2 Binary Function Name Recovery
Different from generating summary for decompiled function, recovering function name requires more accurate understanding about program contexts and more concise abstraction for program semantics. This assists reverse engineers in efficiently navigating numerous functions in a real world binary program. detailed example is provided in Appendix
Setup We use the source code function names as ground truth for name recovery. Similar to binary summarization task, we conduct experiments with three setups: prompting recoverers with only the decompiled code, with decompiled code and source code snippets obtained by retriever, with decompiled code and source code snippets generated by ProRec. The first two setups considered baselines for comparison.
Metrics We evaluate the performance of a tool for the binary function name recovery task at different levels of granularity.
Token-level Metrics. line with existing work in reverse engineering 30), we tokenize both the predicted function name and the corresponding ground truth, then compute precision, recall, and Fl score at the token level. For each metric, we first calculate the scores for individual function name predictions and then average them across all functions.
Character- Metrics. We adapt BLEU 48 METEOR 8 ROUGE-L for the function name by tokenizing function names into characters and computing these metrics on character level, similar to 40 57 They provide fine-grained evaluation of the function names and can avoid some limitations of tokenization.
Results
In all the following experiments, we report ProRec results based on CodeLlama-34b (4bit quantized). For both the retrieval-augmentedd baseline (+retrieval) and ProRec (+ProRec), we use their top-5 contexts as augmentation. The versions of the black-box LLM recoverers are gpt-3.5-turbo- tettuuuro1000 for GPT3.5-turbo, claude-3-haiku. - 20240307 for Claude-3, gemini-1... 0-pro for Gemini-Pro. and gpt- ttr-tturoo2220 2000--224--- 09-09 for GPT4 Evaluator.

---

1000
Dec-only
800
w/ Ret. ProRec
600
400
200
2
1
3
5
Context Relevance
Functionality
Figure 4: Scores from our proposed GPT4 evaluator for summaries generated basd on GPT3.5-turbo. The x-axes denote context relevance (left) and functionality (right), respectively. Larger scores are better. Bars denote the number of summaries with the corresponding score, and dashed lines denote the number of summaries with at least the corresponding score.
4.1 Binary Summarization Results
We show the results for binary summarization Table Observe that ProRec helps all models generate better summary in terms of CHRF. A retriever, the other hand, may introduce noise (irrelevant source functions) to model and even makes the results worse (e.g., for the Gemini- Pro model). Moreover, GPT4Evaluator on functionality (G4-F) and context relevance (G4-C), indicating the summary of ProRec more helpful to human reverse engineer.
see that ProRec achieves higher scores when evaluated with the
We further analyze the results of GPT4Evaluator to illustrate the advantage of ProRec. The results for summaries generated by GPT-3.5 are visualized in Figure 4 It is worth-noting that define the score as a "neutral" score, meaning that a summary does not contain specific context (for the context relevance question) contains only correct but low-level operations without high-level abstractions (for functionality question). We can see that for most cases, GPT-3.5 achieves a score with at least 3. That indicates the LLM can largely understand the low-level behaviors of decompiled code. That is because decompiled code is in the C-syntax.
Table 2: Human evaluation of binary summarization results w.r.t. context relevance and functionality.
On the other hand, we can see that for the context relevance ques- tion, both retrieval -aggeeete baseline and ProRec introduces more useful context information the model, and thus the resulting sum- maries have closer relevance to the ground truth source code. Espe- cially, queries with the code snippets generated by ProRec achieve more scores and 5 than queries enhanced with a retriever's results. That illustrates ProRec indeed generates code snippets with better context relevance than a retriever.
Cont. Rel. Func.
4.29
4.22
+retrieval +ProRec
4.49
4.43
4.76
4.62
For the functionality question, can observe similar patterns. That indicates better contexts introduced by ProRec help the LLM to understand code functionality. We show detailed example in Appendix
Human Evaluation Table 2 presents the human evaluation results from our user study for 50 randomly sampled summaries of each method (using GPT-3.5-turbo as the recoverer). The human evaluation aligns with the automatic metrics: ProRec consistently outperforms the other approaches in terms of both context relevance and functionality, according to human judgment.
4.2 Binary Function Name Recovery Results
We show results for binary function name recovery in Table I We can see that the code snippets generated by ProRec helps all three LLMs predict better names in terms of token-level precision, recall, and F-1 score designed for reverse engineering task (30]. Especially, ProRec outperforms retriever by large margin, indicating that ProRec generates more relevant code than retriever. For character-level metrics, ProRec shows similar improvements, not biasing towards certain metrics. Note that different L.LM recoverers can have different performance on the two tasks, e.g., Gemini-Pro better at binary function name recovery than summarization compared to other two models, yet this does not influence the improvement ProRec brings to both tasks and all recoverers.

---

w/ analysis
w/o analysis
0.18
Token-leve Score
Dec-only
Ret.
0.16
ProRec
0.14
0.12
Top-k
Figure 5: Binary function name recovery results with and without LLM's internal analysis by using top-k additional contexts on 100 examples.
Table 3: Statistics of prober with different base SCFM sizes.
Base SCFM
Trainable Params Ratio (%)
Eval Loss
N-gram Recall (1-4)
CHRF
CodeLlamma77b
27M 37M
0.393
0.6756
27.22 13.69 8.21 / 5.14
31.52 (+0.642)
CodeLlama- 13b
0.283
0,6387
27.51 13.88 8.40 5.32
32.01 (1.0.86)
CodeLLama-34B
80M
0.237
0.5786
27.58 14.06 8.55 5.45
31.54 (1.0.133
5
Analysis
How does black-box LLM's internal analysis A help robust recovery? We study the influence of LLM's internal analysis by evaluating retrieval-augmennt
recovery and ProRec with different number of additional contexts, since we believe this kind of internal analysis is crucial for LLM-based recoverers perform robust binary recovery with additional contexts, especially when the provided contexts are noisy. We run binary function name recovery for 100 randomly sampled examples, times, for zero-shot recovery (dec-only), eetrieval-augmented recovery (recovery), and ProRec. As shown in Figure 5 the internal analysis consistently reduce the variance of function name recovery augmented performance of both regreeeal-aggmmnted recovery and ProRec. This is particularly true for retrieval when & gets large. We deem that it may due to lack of function-level similar source code in the data store. On the other hand, we observe sometimes LLM tend be too conservative without leveraging the extra contexts with the internal analysis, potentially because of our not specifically optimized prompts which can be fixed by making some adjustments. Moreover, we argue that misleading is worse than not informative, and reverse engineers can further interact with LLMs for more aggressive induction after obtaining a conservative response.
Ablation Study on Base Source Code Foundation Model Size ProRec's performance relies on the ability of the base SCFM, where size is a crucial indicator since knowledge is represented by model parameters. Therefore, we study the influence of base SCFM size. We train three probers based on CodeLlamma-7b, Code 13b, and We report statisties of these three probers Table As shown in the table, with growing number of base model size, the prober achieve lower loss on validation set, which leads to an increase in average n-gram overlap of probed source code fragments and the oracle source function, which we run 3 times on 100 examples for each row. However, n-gram overlap with oracle source function seems not significantly influence downstream task performance like CHRF for binary summarization. hypothesize that this potentially due the tasks like binary summarization is not very sensitive to subtle symbolic difference, which means we can leverage modest size SCFM for probing instead of large ones, being economic in real practice.
all in 4bit quantization for fair comparison.
Case Study We examine three specific cases to illustrate the performance and limitation of ProRec Appendix F
Related Work
Large Multimodal Models Recent advancements in vision-language models have demonstrated
their efficacy across range of practical applications such as image captioning [70], visual question

---

answering [5] 16 3 and image-text matching [38. While the limited availability of datasets that align different modalities was perceived as major impediment to scalability, recent works leverage
the knowledge embedded within pre-trained large language models 2 6
37
15
41
80].
Beyond
their capacity to interpret diverse information modalities such as images G3 and audio 225 L.LMs
have increasingly been aligned with graph structures [ 59 and gained widespread attention. In particular, there have been successful attempts that leverage LLMs for graph data involves the Graph2Text strategy, which transforms graph data into textual representations. This technique has been effectively utilized in several studies [6]] |1 A The designs in the prober of ProRec share some similarity with recent LMMs, with a modality -speciffi encoder, and SCFM decoder. However, we tackle the binary-soure code moltiit which is largely unexplored compared to popular modalities. Also, the multi-mode prober is used in a larger probe-and-recee framework instead of end-to-end training.
Retrieval-Augmennte Generation Retrieval-augmented generation is widely applied in knowledge- intensive scenarios, such as question answering [22] 26 31], molecule generation [66], and source code generation [79]. By leveraging a non-parametric datastore, vetrieval-augmente
approaches decompose knowledge and LMs, can complement some long-tail knowledge or keep the knowledge up-to-date without heavy tuning the model which is costly. ProRec, the other head, tries to exploit knowledge within a parametric SCLM for black-box LLM-based binary recovery. A closely related work is GENREAD [76] that prompts InstructGPT to generate context instead of retrieval for knowledge intensive tasks. ProRec differs from this work in that binary recovery requires cross- modal understanding and informative contexts cannot be obtained by directly prompting LLMs We introduce specially designed cross-modal alignment to allow informative context generation.
Binary Reverse Engineering Advances in machine learning models have been widely used to solve 7 588 (0). However, most work focuses on reasoning binary program, and is not human-oriented. Another stream of work trained smaller end- to-end models for individual human oriented tasks, such as variable name prediction [47 [3 72 73).
challenging tasks in binary program analysis [5T 49 61
and function name recovery [30). Nonetheless, these models are not benefiting from pretraining
efforts and thus have sub-optimal performance [56]. Preliminary study shows HOBRE remains a
challenge for state-of-the-a LLMs [29 56). Our efforts attempt to address this challenge, leveraging
pretraining knowledge of SCFMs to help HOBRE.
Conclusion
In this paper, we introduced novel prohe-and-recove framework, ProRec, designed to bridge the semantic gap between binary code and human- undraasaaddet source code. By integrating an aligned binary-source encoder-decoder model with black-box large language models, our approach effectively synthesizes symbol-rich code fragments from binary input, providing valuable context for improving binary analysis tasks. Our extensive evaluations demonstrate that ProRec significantly enhances performance in both binary summarization and binary function name recovery tasks.
Limitations & Future Work We experiment with simple achitecture and straightforward align- ment of the binary-source prober this paper, which might not be optimal for ProRec. Future work can explore better prober architecture and alignment objectives. Moreover, currently we only focus on intra-proe prore-proeedure analysis, similar to most existing work. In practice, HOBRE needs to deal with full binary with multiple functions. An important direction will be extending ProRec to pnoeeuprocedure scenarios, where additional information from the whole program such as call-graph can be leveraged, building program-level binary reverse engineering agents.
8
Acknowledgement
We thank the anonymous reviewers for their valuable comments and suggestions. We thank all the participants in our user study. We are grateful to the Center for AI Safety for providing computational resources. This research was supported in part by DARPA VSPELLS HR001 120S0058, IARPA TrojAl N911NFFI19SS001 and N00014171240 Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
NSF 290 and 1910300, ONR N00014171 12455, NO001 41410468
10

---

References
[1] Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen. Extending source code pre-trained language models to summarise decompiled binarie. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 260-271, IEEE, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. beiio-anddlanguage navigation: Interpreting usuaaly-groundeer navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018
[4] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint 2.XXxx:330..00000 2023.
[5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering, In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[6] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre. Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint 0rXiv:2308.01390 2023.
[7] Pratyay Banerjee, Kuntal Kumar Pal, Fish Wang, and Chitta Baral. Variable name recovery in decompiled binary code using constrained masked language modeling. arXiv preprint ar arXiv:2103. /2801, 2021.
[8] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In P Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation andfor summarization, pages 65-72,
2005.
[9] Adam R Bryant. Understanding how reverse engineers make sense of programs assembly language representations. Air Force Institute of Technology, 2012.
[10] Kevin Burk, Fabio Pagani, Christopher Kruegel, and Giovanni Vigna. Decomperson: How humans decompile and what we can learn from it. In 31st USENIX Security Symposium (USENIX Security 22), pages 2765-2782 2022.
[11] Ziwei Chai, Tianjie Zhang, Liang Wu, Kaigiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint 10X.v:2310.05845, 2023.
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint X0X::2007.03374, 2021.
[13] Qibin Chen, Jeremy Lacomis, Edward J. Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security 22), pages 4327-4343, Boston, MA, August 2022. USENIX Association.
[14] Qibin Chen, Jeremy Lacomis, Edward Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security 22), pages 4327-4343, 2022.
11

---

[15] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[16] Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326-335, 2017.
[17] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers scale. Advances Neural Information Processing Systems, 15:30318-33 1--3338--3332,, 2022.
[18] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuming of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[19] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyrr and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.059991 2022.
[20] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024.
[21] Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305. 15066, 2023.
[22] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning. pages 3929-3922222 PMLR, 2020.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-97335 923899978, 2020.
[24] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
[25] Rongjie Huang, Mingze Li, Donghhao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23802-23804, 2024.
[26] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, Online, April 2021, Association for Computational Linguistics.
[27] Ling Jiang, Junwen An, Huihui Huang, Qiyi Tang, Sen Nie, Shi Wu, and Yuqun Zhang. Binaryai: Binary software composition analysis via intelligent binary source code matching, In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages
1-13, 2024.
[28] Nan Jiang, Chengxiao Wang, Kevin Liu, Xiangzhe Xu, Lin Tan, and Xiangyu Zhang. Nova + Generative language models for binaries. arXiv preprint arXiv:2311.13721, 2023.
[29] Xin Jin, Jonathan Larson, Weiwei Yang, and Zhiqiang Lin. Binary code summarization: Benchmarking chacgp//gtt-t and other large language models. arXiv preprint arXiv:2312.09611 2023.
12

---

[30] Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. Symlm: Predicting function names in stripped binaries via context-sensitive execution-awar code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 1631-1645, 2022.
[31] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769 6781, 2020.
[32] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barsskka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(L3):3211-352 2017.
[33] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, al. The stack: 3 tb of permissively licensed source code. arXiv preprini arXiv:221 2022.
[34] Jeremy Lacomis, Pengcheng Yin, Edward Schwartz, Miltiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Dire: A neural approach to decompiled identifier naming. In 2019 34th IEEE/ACM International Conference Automated Software Engineering (ASE), pages
JEEE, 2019.
[35] Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwarle al. Standing on the shoulders of giant frozen language models. arXiv preprini urXiv:2204. 10019, 2022.
[36] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et eetrieval-augmentt generation for pnowledge-intensing nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9945 09449999444444 2020.
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19740, PMLR, 2023.
[38] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In Proceedings the vision, pages 4654 4662, 2019.
the IEEE/CVF international conference on computer
[39] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[40] Xi Victoria Lin, Chenglong Wang. Luke Zettlemoyer, and Michael D Ernst. N12bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
[41] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[43] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li. Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu,
13

---

Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang. Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder and the stack v2: The next generation, 2024.
[44] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.0885500 2023.
[45] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Ych, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. Anymal: An efficient and scalable mny-modality augmented language model. arXiv preprint vrXiv:2309.60055 2023,
[46] OpenAl. Gpt-4 technical report, 2024.
[47] Kuntal Kumar Pal, Ati Priya Bajaj. Pratyay Banerjee, Audrey Dutcher, Mutsumi Nakamura, Zion Leonahenahe Basque, Himanshu Gupta, Saurabh Arjun Sawant, Ujjwala Anantheswaran, Yan Shoshitaishvili, et al. "len or index or count, anything but v1": Predicting variable names in decompilation output with transfer learning. In 2024 IEEE Symposium on Security and Privacy (SP), pages 152-152. IEEE Computer Society, 2024.
[48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[49] Kexin Pei, Jonas Guan, Matthew Broughton, Zhongtian Chen, Songchen Yao, David Williams- King, Vikas Ummadisetty, Junfeng Yang, Baishakhi Ray, and Suman Jana. Stateformer: Fine-grained type recovery from binaries using generative state modeling. In Proceedings of the 29th ACM Joint Meeting European Software Engineering Conference and Symposium the Foundations of Software Engineering, pages 690-702, 2021.
[50] Kexin Pei, Jonas Guan, David Williams-King Junfeng Yang, and Suman Jana. Xda: Accurate, robust disassembly with transfer learning. In NDSS. The Internet Society, 2021.
[51] Kexin Pei, Zhou Xuan, Junfeng Yang, Suman Jana, and Baishakhi Ray. Trex: Learning execution semantics from micro-traces for binary similarity. arXiv preprint arXiv:2012.08668 2020.
[52] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-ICCNIP) pages 2463-2473, 2019.
[53] Maja Popović. chrf: character n-gram f-score for automatic evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392-395, 2015.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 3748-8763. PMLR, 2021.
[55] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. 2024.
[56] Xiuwei Shang. Shaoyin Cheng, Guoqiang Chen, Yanming Zhang, Li Hu, Xiao Yu, Gangyang Li, Weiming Zhang, and Nenghai Yu. How far have we gone in stripped binary code understanding using large language models. arXiv preprint arXiv:2404.09886 2024.
14

---

[57] Freda Shi, Daniel Fried, Marjan Ghazvininejad. Luke Zettlemoyer, and Sida I Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533-3546, 2022.
[58] Zian Su, Xiangzhe Xu, Ziyang Huang, Zhuo Zhang, Yapeng Ye, Jianjun Huang, and Xiangyu Zhang. Codeart: Better code models by attention regularization when symbols are lacking. Proceedings of the ACM on Software Engineering, 1(FSE))562-55) 2024.
[59] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.1302. 2023.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:230..09288 2023.
[61] Hao Wang, Wenjie Qu, Gilad Katz, Wenyu Zhu, Zeyu Gao, Han Qiu, Jianwei Zhuge, and Chao Zhang. jtrans: Jump-aware transformer for binary code similarity. arXiv preprint
1mxiv:2205.127133 2022.
[62] Heng Wang. Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Informa- tion Processing Systems, 36, 2024.
[63] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
[64] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint 2..iv:2300.07922. 2023.
[65] Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-awar unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021.
[66] Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anand- kumar. Retrieval based controllable molecule generation. In The Eleventh International Conference on Learning Representations, 2022.
[67] Jiayi Wei, Greg Durrett, and Isil Dillig. Coeditor: Leveraging contextual changes for multi-round code auto-editing. arXiv preprint FrXXi:2305.1888. 2023.
[68] Shi Weijia, Min Sewon, Yasunaga Michihiro, Seo Minjoon, James Rich, Lewis Mike, al. Replug: Retrieval-augmented black-box language models. ArXiv: 2301.12652. 2023.
[69] Jiaqi Xiong, Guoqiang Chen, Kejiang Chen, Han Gao, Shaoyin Cheng, and Weiming Zhang. Hext5: Unified pre-training for stripped binary code information inference. In 2023 38th IEEF/ACM International Conference on Automated Software Engineering (ASE), pages 774. 786. IEEE, 2023.
[70] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[71] Xiangzhe Xu, Shiwei Feng, Yapeng Ye, Guangyu Shen, Zian Su, Siyuan Cheng, Guanhong Tao, Qingkai Shi, Zhuo Zhang, and Xiangyu Zhang. Improving binary code similarity transformer models by semantics-driven instruction deemphasis. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, page 1006-1 1118, New York. NY, USA, 2023, Association for Computing Machinery.
15

---

[72] Xiangzhe Xu, Zhuo Zhang. Shiwei Feng, Yapeng Ye, Zian Su, Nan Jiang, Siyuan Cheng, Lin Tan. and Xiangyu Zhang. Lmpa: Improving decompilation by synergy of large language model and program analysis. arXiv preprint urXivv2306.0054400 2023.
[73] Xiangzhe Xu, Zhuo Zhang. Zian Su, Ziyang Huang, Shiwei Feng, Yapeng Ye, Nan Jiang, Danning Xie, Siyuan Cheng, Lin Tan, et al. Leveraging generative models to recover variable 2023.
names from stripped binary. arXiv preprint arX.::2330.0223350
[74] Ruosong Ye, Caiqi Zhang, Runhui Wang. Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.0711 2023.
[75] Tong Ye, Lingfei Wu, Tengfei Ma, Xuhong Zhang, Yangkai Du, Peiyu Liu, Shouling Ji, and Wenhai Wang. Cp-bes: Binary code summarization guided by control flow graph and pseudo code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14740-1477 2023.
[76] Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chen- guang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. The Eleventh International Conference on Learning Representations, 2022.
[77] Yuexiang Zhai, Shenghang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arX9v:2309. 10313, 2023.
[78] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[79] Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. Doeprompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2022.
[80] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing rision-languaa understanding with advanced large language models. arXiv preprint arXiv:2304.10522 2023.
16

---

A Details for Training Assembly-Souree Code Dual-Encoder
We discuss details of the contrastive training of the assembly-souree code dual-encoder in this section. Given a mini-bate batch B {< + with batch size N, where x. represents the i-th tuple of assembly code and its dependency graph, zi represents the corresponding i-th source code, we train the dual-encoder, g(-) for binary encoder, and h(()) for source encoder, with the following objective
Ledua- = 1 (fazs + fazz))
(4)
2
where
exp(sim h(z)))
Lass
log
(5)
2j=1 exp( sim (z+)) h(z3)))
and
N
log (=p(sim(((x;),...)) h(;]),
E2
(6)
==1 exp(sim(g(e)) (g())) h(x))))
Here, we use cosine similarity sim(...). )). In order to have more negative samples, we use momentum encoders (3 for both modality with a queue size 4096 and momentum 0.999. We train the model with learning rate 5e-5, a batch size of 16, Ik warmup steps, and 17k total steps.
The aligned dual-encoder as a cross-modal dense retriever achieves 84% recall@) on the validation with pool size 10k, which demonstrates that it has reasonable retrieval performance.
B Details in Experiment Setup
B.1 Dataset Quality Assurance and Preprocessing
We ensure data quality by (1) selecting projects with no less than 20 stars, (2) including only executable binary programs that can be fully stripped, and (3) deduplicating our dataset by checking the source code string-
Initially, we obtained 18k projects from Github. We tried to compile all of them in x86-64 with O0, and discarded not compilable ones. generates 106k executable binaries. We then match binary functions with source code functions by their function names, and deduplicate data samples by their source code strings (e.g., some utility functions may be used in multiple programs). Our final dataset containing 270k pairs of binary and source code functions.
B.2
Evaluation Aspects and Rationale for Human Study and CPP4Evaluator
We provide GPT4 with the decompiled code, the corresponding source code, and the reference summary evaluate. For each question, we adapt the Likert scale I and instruct GPT4 output score from (worst) to 5 (best). The users in our human study are provided with similar information as questionnaires to perform human judgment of summaries.
We derive our evaluator prompts from a thorough survey on reverse engineer 19. summarizes 8 sub-goals of human reverse engineers. We list them in Table 4 and categorize the goals into four scopes. We highlight ones that binary summarization can help.
The survey
Specifically, for goal (7), reverse engineer aims to reason about the high-level abstraction of the program, e.g., what the program does, and how the program works (.. We use the prompt in Figure to evaluate how helpful a summary is obtain the high-level picture of program.
17

---

Table 4: Goals in reverse engineering. We construct the table from a thorough study for human reverse engineers (Talle 12).
Scope
Goal
(1) Understand the purpose of analysis
Related to specific analyses
Finish the analysis quickly
(3) Discover general properties of the program
Easy access
(e.g., size of the program)
(4) Understand how the program uses the system interface
Addressed by the decompiler
(5) Understand, abstract, and label instruction-level information
(6) Understand how the program uses data
Construct complete "picture" of the program
Can be enhanced by summarization
Understand, abstract, and label the program's 's functions
A. Does the summary reflect relevant context (domain)? Answer the question in range 5(best) to 1(worst). Domain/context describes the purpose of function. is more of the general high-level domain (e.g, network, memory, CPS, physics, GUI, etc) rather than specific functionalities (e.g.. sort, string comparison, memory allocation).
For 5, summary the reference should describe the same domain/context.
For 4, the domain of the summary and the reference should be similar and relevant, although may not be exactly the same. The summary domain may be superset subset of the reference. The summary domain may be closely related to the reference domain. The summary and reference may be two different perspectives of same specific domain.
For 3, the summary does explicitly mention specific context. It only contains low level operations. From the summary, one cannot deduce the high-level purpose of the decompiled function.
For the summary is slightly misleading. The summary domain is different and not relevant to the reference domain. However, it is implied by the choice words in summary, and not explicitly mentioned.
For 1, the summary is completely misleading. The summary domain is irrelevant to the reference domain, and is explicitly mentioned in the summary.
Your output should first briefly comment the summary from the aforementioned perspectives. Do not allow the length of the responses influence your evaluation. Be as objective as possible.
Figure 6: Prompts for GPT4-Evaluator for asking context relevance.
For goal (8), a reverse engineer reasons specific behavior individual functions to form the mental models (9] of the program logic. We the prompt in Figure to illustrate how accurate summary is to describe the functionality of a program.
For other goals in Table 4 goals (1-2) are associated with specific analyses, instead of programs. Goal (3) aims to capture the general properties of a program (e.g., the size of program, the sections in binary executable file). These properties are easily accessible. The following three goals (4-6) are achieved by a decompiler. The decompiler recovers call to the system APIs (goal 4), reasons instructions and lifts them to C-like syntax (goal 5), and recovers data dependences by introducing variables (goal 6). Therefore, the focus of binary summarization is on the last two goals, requiring understanding and reasoning of program semantics.
B.3 Importance and Use Scenarios of Function Name Recovery
Function name recovery is important to the reverse engineering task because a human typically starts the reverse engineering task by achieving rough understanding about all functions, as suggested by studies on human reverse engineers 10 For example, malware sample to analyze may contain hundreds of binary functions. A reverse engineer will need to first locate functions with suspicious
18

---

B. Does the summary reflect relevant functionality? Answer the question in range 5(best) to I(worst). Functionality means the specific high-level behaviors performed in function (e.g.. sort, string comparison, decoding package, printing error messages).
For 5, the functionality in the summary should be almost exactly the same to the reference.
summary are similar the reference. It may be vague in details, but the overall functionality and purpose is correct.
For 4, the functionalities
For 3, the summary does not specify functionality. It only repeats some low-level operations without high level abstractions.
For 2, the summary specify relevant but inaccurate functionality, The functionality specified in the summary may be relevant the reference summary, but they have significant differences.
For I, the summary contains irrelevant functionality. is contains totally different hehavior with the reference.
Your output should first briefly comment the summary from the aforementioned perspectives. Do allow the length of the responses influence your evaluation. Be as objective as possible.
Figure 7: Prompts for GPT4-Evaluator for asking functionalityy
behaviors (e.g., executing commands received from a remote server) before analyzing the function in detail. The workload would be huge even if all functions have natural language summaries. On the other hand, all the decompiled functions have names as in the source code, a human developer can efficiently go through the list of function names and identify functions requiring further inspection.
B.4 Formal Definition of Precision, Recall, and F1 Used by the SymLM Metrics
Formally, the token-level precision (P), recall (Rt), and Fl are defined as follows:
T
TTT
T
2 x P() x R()) F1(1)
P(i)
R(i)
T (6)
P(i) + R()
where 2 is the token set of the ground truth name for the i-th test case, and T the token set of the i-th predicted name.
The precision, recall, and F1 scores for the entire test set are the average scores of individual scores across all test cases. Formally,
P P=NZ P
1 R=
1 N R(i)
Σ
F
1 N
Σ Fi(1),
where N is the number of test cases.
C More Experiment Results and Analysis
C.1 Comparison with Supervised Binary Summarization Methods
Different from existing work CP-BCS 75 that supervisedly train model for binary function summarization, ProRec does not require any superivsed training data for summarization and directly rely on LLM's summarization ability. More importantly, CP-BCS's summarization target is the docstring/comment of function parsed from source code, which is not identical as the summarization targets in our experiments which are LLM summarizations from source code. For a fair comparison, we prepend the comments summarized by CP-BCS to the decompiled code additional context for LLMs
1006) to revise it into their own summarization styles, so that the final candidate summaries can be properly compared with reference source code summaries. Here, "+CP-BCS comment" means we augment the decompiled code with the comment for LLM to summarize. If we only evaluate the comments generated by CP-BCS, the CHRF drops 5.44.
19

---

System: You are an experienced C/C++ software developer.
User: You provided with the following function:
{]
First generate a brief stpppy description of its functionality in the format:
Description:
Then generate high-level summary of its functionality in the format:
Summary: The function...
After that, generate brief description its general purpose in the format:
Purpose: The purpose of the function is to
Figure 8: Prompts for source code summarization.
We can see in That CP-BCS comments have negative impacts on binary summarization results (based on gpt3. mpt3.5-ture 1--turbo-110 1106) on our test set, po- tentially due to the distribution difference between training and test data. In fact, cannot easily adapt CP-BCS to this distribution since the training requires comments within the source code which do not exist in many functions in our training data. It is possible distill summarization from LLMs, but the cost is high given the large amount of data. On the contrary, for ProRec data is less of problem since all the compilable projects can be used to produce binary-source pairs that can used for alignment.
Table 5: CP-BCS generated comment-
augmented binary summarization results.
CHRF
G4-C
G4-F
decompiled-only +CP-BCS comment 29.0
30.4
3.6
3.8
3.0
2.8
C.2 Comparison with Black-box LLM Prober
Leveraging black-box LLMs as probers is challenging because they are not heavily pre-trained on binary code and have limited understanding of it. ProRec addresses this through alignment training.
To demonstrate this empirically, we conduct exper- iments binary function name recovery. We first prompt a black-box LLM 5-tt:r.---tb---oo translate decompiled functions into readable ones, sampling multiple results as diverse probed contexts. Using the same prompt ProRec and the same LLM, we perform function name recovery with additional context. We call this method "self- sself-probing"
Table 6:
Binary function name recovery re- sults with self-probing on 100 examples.
to
PSmLIM
Fsyml RSymIM
M
decompiled-only
16.47
19.40
16.79
+retrieval
17.75
22.05
18.72
+ProRec
20.38
26.72
21.84
shows the performance of self-probing (based 3pt3.5-turee 106) compared other meth- ods on 100 randomly sampled test data. We can see that self-probing performs slightly better than direct- prompting but is not comparable to retrieval-augmmentd recovery or ProRec.
+self-probing
16.02
20.52
17.01
D Prompts Used
We show our prompts to generate source code summarization in Figure g the prompts to generate decompiled code summarization in Figure I0, and the prompts to recovery function names in Figure 10 Note that the prompts for GPT4Eyaluator are discussed the previous section, shown in Figure Glnd Figure
E
User Study for Binary Summarization
The user study involved 12 participants. The participants are PhDs / PhD students that either have some background reverse engineering or are experienced in C/C++/Rust programming. We ensured each summary is scored by at least 3 users, and use the median scores the results. The questions in
20

---

System: You are an experienced binary reverse engincer to understand decompiled code that lacks symbol information.
User (Default):
You provided with the following decompiled function that is hardly human readable:
()
First generate a brief step-by-step description of its functionality in the format: Description:
Then try generate summary of it that help human understand / inspect its original high-leve geheleevel source code functionality in the format:
Summary: function...
After that, inspect and generate a brief description of its general purpose in the format:
Purpose: The purpose of the function seems to
User (Augmented):
You are provided with following decompiled function that not human readable:
()
First generate brief step-by-step description of the functionality of the decompiled code in the format: Description:
Then try generate summary of that help human understand / 'isspect its original high-level source functionality the format: Summary: The function...
After that, consider the following source functions (if any) that are potentially relevant to this decompiled function.
[source functions)
Analyze whether they are relevant to the decompiled function the format: Analysis:...
Finally, based the analysis, to inspect and generate the general purpose of the decompiled function
the format:
Purpose: purpose function seems to
Prompts for decompiled code summarization. User (Default) denotes directly prompting. while User (Augmented) denotes prompting with relevant source code snippets obtained by tool.
Figure 9:
System: are an experienced binary reverse engineer to understand decompiled code lacks symbol information.
User (Default):
You have decompiled function from executable, which currently has generic name like sub_ The decompiled function code follows:
()
Generate a more human-understanddble function name for the decompiled code to replace the original sub_xxx in the format:
Function Name: function name _ae__aameeee_______eeeeeeee Lecel__oos__eeeee _eeree
User (Augmented):
have decompiled function from an executable, which currently has a generic name like sub_xxee _xxx. The decompiled function code is as follows:
{)
Consider the following source functions (if any) that potentially relevant this decompiled function. [source functions]
Analyze whether these source functions are relevant to the decompiled function in the format: Analysis:
Then, based the analysis, generate more human-understanddar function name for the decompiled code to replace original sub sub_xx, _xx in the format:
Function Name: function_nsant cacliionnnnee__eeeee www._aameeeeeeeee saaame__oe_____e goee_hereeeeeeeee hheee
Prompts for function name recovery. User (Default) denotes directly prompting, while Figure 10:
User (Augmented) denotes prompting with relevant source code snippets obtained by tool.
21

---

our user study were sampled from summaries generated by all three techniques (i.e., ProRec, the retrieval-uugmented baseline, and direct prompting baseline).
The study leverages the same prompts used in the GPT4Evaluator (details in Appendix B.2 as the questionnaires, asking users evaluate piece of summary in terms of context relevance and functionality, As we already show the results of human judgment of different approaches in the main text, here we only show the results of Spearman correlation between human judgment and automatic metrics in Table 7 We can see that both CHRF and GPT4Evaluator are consistent with human preference. with GPT4Evaluator the most consistent metric with regard to human scores. Therefore, we use CHRF and GPT4Evaluator to evaluate the quality of binary summarizations.
Table 7: Spearman correlation between human preference and auto-metrics. Columns 2-3 and 4-5 are for the context relevance questions and functionality questions, respectively. For each question, report both the correlation and the p-value. A higher correlation value and smaller p-value indicate a statistically stronger correlation. Bold and underlined text indicate the best and second-best values, respectively, in each column.
Context Relevance
Functionality
Metric
Correlation
p-value
Correlation
p-value
METEOR
0.51
5.4e-5
0.39
2.5e-3
BLEU
0.28
0.05
0.21
0.11
ROUGE-L
0.49
1.1e-4
0.40
1.9e-3
CHRF
0.56
4.7c-6
0.48
1.5e-4
GPT4Eval.
0.58
2.7e-6
0.58
2.6e-6
F
Case Study
In Figure [1 we show function that initializes a key for encryption. Without any context information. GPT-3.5 summarizes the function with generic descriptions (e.g., "maniplate and transform the input data"). On the other hand, ProRec generates code snippets (only one of them is shown here) related to encryption keys. Provided with these code snippets, GPT-3.5 correctly summarizes the function as "perform cryptography operations", and mentions several operations related to "key". Although the summarization does not perfectly reflect the "initialization" purpose of this function, the description clearer and more relevant to the context (i.e., key operations). We can see that retrieval results helps LLM generate more context-relevant yet not functionally correct summary. That is because the datastore contains code snippets that are very similar to the query function (e.g., the function 11__256_ec__reee in Figure II (d) is crypto-related function that performs bit-wise operations). However, the retrieved function does not explicitly mention anything like "key", which prevents the LLM recoverer to further guess the purpose of the function.
Figure 12 shows more extreme case when RAG less helpful than ProRec with no relevant function similar to the query function in the datastore. The query function pops an element from queue. The retriever retrieves two snippets of code that have similar syntactic features (e.g.. null pointer checks at the beginning; pointer accesses in the loop condition). By contrast, ProRec recognizes local semantic information such as getting an element from queue, and atomic memory operations. Therefore, the probed code snippets are more relevant to program contexts even if the entire query function is not in the datastore.
We study a failure case of ProRec for the function name recovery task. The example is shown in Figure 33 The function reads data from temperature sensor and convert the temperature from raw sensor data to human-readabe temperature unit. We show two code snippets generated by ProRec. We can see that ProRec successfully recognizes the function is relevant to read and parse data from sensor. However, it does not accurately associate it with the temperature sensor. Therefore, although generated summary is of better quality, the recovered function name is still different from the source code.
22

---

int64 fastcalll sub_182282(
void orseferpInittccnns unsigned charskey, syrmetric_Ke eskey!
static unsigred char aod_tablel] (/x ... 1//3 Ix x--)
124
_YTE +lvd BBYEE +)(1 all:
slunsigned #2(1 all:
keyixl;
key(xl:
v4[0]; v3 v4[1];
v4[al: +ix
ix y: senppytskyy.ssaarrp... x
for (i
w(t_DYTE slvd
(8 +lv4
5)
tlyl ((rly))) 255;
result tunsigned intli;
v8
++1
ly
int64/6222
skey
byte 485440
bfasbasss(x-1y) 255; (ooodd table
+byte_ 4051_44512001
333
}}}
(a) Decompiled Code
(b) Source Code
void aes_key_eexpansion
2o__25__ccc_reee (...)(
char
// ... for (i=0; ic43; i++) {
unsigned char temp[d];
= [int8_tin;
for (1 wiil keylil;
1++1
y & 6x3f; n >>= 6;
for (1 16; e 176; 14+)
for (1 05 1991
temp wii
11
(c) Probed Code
(d) Retrieved Code
Summary w/ only decompited code: The purpose of the function secms to be to manipulate and transform the input data located at momory address and the result the memory locations pointed az"....
Summary w/ retrieved code: The purpose of the fimction seems to involve processing input elements iteratively. performing bitwise openaions, and memory manipulations produce new output army. It likely implements specific algorithms for data mansformation and manipulation.
Summary probed code: The purpose of the function seems to be to perform cryptographic operations, It likely involves key expansion, input enerypted or decrypted
substirution, permutation, other cryptogrphic operations
(e) Summaries
Figure 11: ProRec helps generate summary that is more relevant the source code context.
23

---

.int64 fastcall sub 181696(volati) signed Int32 *al, vold +a2]
int 1f [f_queue_pop(C poop[Lf_quuue _qqueee queue, void #unit)
//
(Iqueue lunit) return
[ial (a2)
read the top
do(
dof
= +1a1 21;
P_head head-op_ head;
(signed int32 *)i(char *]a1 (v4 (+a1 281:
i_head UNIT SWIT_HÉADIC (ETT_HEADQqqeee, p_head);
while sy__cmc__oo boo compare and swapssswpppcc 6head->p_ sp__--d,,,,,p---pp p_head, u u_hhad-onexti);
while [v4 l= InterlockeeoooprrExxmnnnlllt1 *v3, v41);
read nencpy(unit, UNIT DATA- u. henee)))eext,,,
nencpyía2, Ichar +)a1 (+v3 1)
(+al + 8) 36, +all; *1 Kichhr +)a1 (*v3+1) (=a1-8)-32)= 0;
update the head-ounit_sizel; UNIT_ (QUT_,,AD v_heed-onext)-xveeeflaa false;
return 0;
return 8;
Decompiled Code
(b) Code
Int getlleeeettttteee Queuee voidk data) { !data) return
reg_errcode_t re_node__ f...M
if (src NILL src-onelen return REG NOERROR; (dest->alloc arc-onelen dest-onelen len))))))
struct queue_s* queve Istruct queue_s*1qp
rwlock read read_unlock(.... (1rwlock irdlock) struct rwlock rwlock_s
int mpi_copy( *x, *Y 0;
fort Yoon - 1; 85 i--
= (struct mitock_se/milock;
ift Y-splil t= 0
break;
(c) Probed Code (Top 2)
(d) Retrieved Code (Top 2)
Summary w/ decompiled code only: The purpose of the function seems to involve managing and transferring data in thread-safe manner, likely for inter process communication synchronization. Fune name decompiled code only: process_ _add lal___aandd_____eeee
Summury probed code:
data element from queue-lile like data stracture and copy provided memory location while ensuring thread safety through use atomic operations.
The purpose the function
Fune probed code: queue dequeue
Summary retrieved code: The purpose the function seems to involve atomic memory operations for manipulation copying data. It may be part of'a larger system that deals with concurrent access and modification of shared memory. Fune name retrieved code: memory_man ip Lat iii_iinnnnnn and operoppteiion
HOBRE Results
Figure 12: ProRec can be more helpful than RAG in HOBRE tasks when no relevant function can be retrieved from the datastore.
24

---

Lint54 _ffastalll sub_4811339 .ntt4 al, .MRRO +az, float +031
uintä_t htuzld_read, heaelrrrrddddeeteeeeeeeeeeeeeeee terperrauuureetuu21 handle_t shandle, wintis_T stesperature, raw, float +texperature_s) if (handle == NULLIC /x check handle 4/ return 2;
if (1e1) return 2L1:
if (+T_ +llal 72)
if (eandle- = HTUZID MODE HOLDHOLDD MASTER(( res a_htu21d, a_hru2ld_rradd.... readt... / read tesperature +/
v5 -13;
v1e 1012440202211,, C int541555, lul:
// *tenperature_ date;
//... v9:
Je convert raw tesperature */ stengerature_s (etorprrtto rawl/65536.0
+33 (175.72 ((1laat)=a2
65536.811 811
46. 04999;;
175.72f 46.05f;
return BLL;
return 8;
(a) Decompiled Code
Code
uinta_t sht4x_get, hun.ddty(sssdyy handle_ 1 whandte, uint16_t sra,, flost shit if Chandle NULLI /s check handle */ return 2;
Summary w/ probed code: The purpose of fumction seems be to process data hased specific conditions and input panimeters, possibly related to environmental sensor reading,
1f (handle xinited 1-
Generated Summary
uinta_t sap www.tttccc__aanneeee shandle, uiarie_r etvne, appb)
if (handle -- MILLM /e check handle +/ return 2;
if (handle->in
sensor_data_pe _att__proessiig
(c) Probed Code (2 Snippets)
(c) Recovered Function Name (different with source code)
Figure 13: ProRec helps generate summary that is relevant to the source code context.
However, the recovered function name is different from source code.
25

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Please refer Section 7
26

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper contains no theoretical result. The approach is evaluated empirically.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the information needed to reproduce the main experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our code and data are available at htttps///gituub.com/iaaaassss
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http://oeeubiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
28

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the detailed experimental setups.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We draw confidence intervals in our plots, report p-values for user study, All other experiments are conducted on relatively large dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We use LLMs that are public available. Our training is conducted using NVIDIA A100s.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the code of ethics and confirmed that we do not violate the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No. Our work focuses specific domain of reverse engineering.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended
functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
30

---

Justification:
This paper does not introduce particular safety risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We have described the data and tools used for the evaluation in Section ?? which are open-sourced and granted by the authors.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwowethhde...m caree.com/eee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
31

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We conduct user study and obtained IRB approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "6lEFE9Oihh", "reviewer": "Reviewer_dbwT", "summary": "This paper presents a new framework, using an encoder-decoder architecture, call ProRec and an LLM black-box model for helping convert binary code in human readable format. The authors try multiple models in an effort to develop ProRec, and settle on using CODEART and Codellama. For black-box LLM they experiment with state-of-the-art models like GPT3.5, Claude-3 and Gemini-Pro.\nThey generate their own train and test data from GitHub and test on two tasks, summarization and function name recovery. Results improve significantly by using ProRec.", "questions": "- Which other fields could benefit from the ProRec architecture you propose?\n\n- How do you plan to extend this work in the future?", "limitations": "There is no limitations section in the paper. However since they deal with the security domain, they should consider how it may be used by malicious actors.\nIs it possible that the work is used by malicious agents to understand better and exploit critical software infrastructure? \n\nAnother limitation that could be addressed is the impact of the work. To me it seems limited in terms of application to other domains.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper targets a very interesting and important problem area. It is a well written paper proposing a novel architecture for making binary code more human readable.\n- The authors experiment with multiple code models before settling on CODEART and Codellama for ProRec.\n\n- They use multiple state-of-the-art black box models as baselines. They also use a retrieval augmented baseline.\n\n- They results look promising for both tasks across all the models.", "weaknesses": "The only major weakness that I could see were in-terms of impact of the given approach. \n\n1. Applicability to other domains:\n- The proposed architecture for generating human readable forms of binary code seems very customized for the problem. It is not clear how the given approach can be useful for any other domain apart from the one mentioned.\n\n2. Applicability to other tasks in the domain:\n- Since I am not familiar with this particular domain, it is not even clear to me which other tasks beyond function name generation and summarization can benefit from this approach. If the authors feel they can extend the work with more tasks, they should mention it in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your kind words! We are delighted to know  that you enjoyed the well-grounded motivation and thorough experiments.\n\n> ### Q1. Security concerns about ProRec being used by malicious agents to understand better and exploit critical software infrastructure\n\nJust like LLMs could improve malicious agents’ productivity, our method might help them too. We will be careful on the license and regularization when releasing our models and data.\n\n> ### Q2. Which other fields could benefit from the ProRec architecture you propose?\n\nHuman-oriented binary reverse engineering is a fundamental research field in software security. ProRec can potentially benefit security assessments of contemporary devices such as home automation systems [1, 2] and autopilot technology [3], maintaining and hardening legacy software [4, 5, 6], detecting vulnerabilities in commercial-off-the-shelf software [7, 8], and analyzing malware [9]. We will add a discussion in revision.\n\n[1] Angelakopoulos et al. {FirmSolo}: Enabling dynamic analysis of binary Linux-based {IoT} kernel modules. USENIX Security 23.\n\n[2] Aonzo et al. Humans vs. machines in malware classification. USENIX Security 23\n\n[3] Miller et al. Probabilistic disassembly. 2019 ICSE.\n\n[4] Carlini et al. {Control-Flow} bending: On the effectiveness of {Control-Flow} integrity. USENIX Security 15.\n\n[5] Martin et al. Dynamically checking ownership policies in concurrent C/C++ programs. 2010 ACM Sigplan Notices.\n\n[6] Carbone et al. Mapping kernel objects to enable systematic integrity checking. 2009 CCS.\n\n[7] Xu et al. Spain: security patch analysis for binaries towards understanding the pain and pills. 2017 ICSE.\n\n[8] Li et al. SemHunt: Identifying Vulnerability Type with Double Validation in Binary Code. 2017 SEKE.\n\n[9] Xu et al. Autoprobe: Towards automatic active malicious server probing using dynamic binary analysis. 2014 CCS.\n\n> ### Q3. How do you plan to extend this work in the future?\n\nWe plan to extend this work in the following directions in the future:\n\n- Building program-level agents that analyze entire binaries containing multiple functions.\n- Applying ProRec to various downstream tasks such as vulnerability detection and software hardening.\n- Integrating it with popular reverse engineering tool chains such as IDA [1] and Ghidra [2] to achieve broader impact.\n\nWe will add a section discussing the limitations and future work in the paper.\n\n[1] IDA-Pro. https://hex-rays.com/ida-pro/\n\n[2] Ghidra. https://ghidra-sre.org/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DaAWcIx3V1", "reviewer": "Reviewer_pcKr", "summary": "Human-Oriented Binary Reverse Engineering (HOBRE) seeks to transform binary code into human-readable content that aligns closely with its original source code, effectively bridging the semantic gap between binary and source. While recent advancements in uni-modal code models, including generative Source Code Foundation Models (SCFMs) and binary understanding models, have been promising, their application in HOBRE has been limited by reliance on either supervised fine-tuning or general large language model prompting. This has often led to suboptimal outcomes. Drawing inspiration from the success of multi-modal models, the authors propose a novel \"probe-and-recover\" framework that synergistically combines binary-source encoder-decoder models with black-box LLMs for enhanced binary analysis. This framework uses pre-trained SCFMs to generate symbol-rich code fragments as context, improving the interpretive capabilities of black-box LLMs (termed \"recoverers\") used in the process. The proposed approach has demonstrated significant enhancements in zero-shot binary summarization and binary function name recovery tasks. Notably, it achieved a 10.3% relative improvement in CHRF and a 16.7% relative gain in a GPT-4-based metric for summarization, along with a 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results underscore the effectiveness of the authors' framework in automating and refining the process of binary code analysis.", "questions": "The paper explores a critical and emerging area within cybersecurity and software engineering, introducing a novel \"probe-and-recover\" framework that effectively enhances binary reverse engineering. It demonstrates marked performance improvements in binary summarization and name recovery, achieving significant gains in established metrics. Additionally, the authors have commendably made the code available as open source, promoting transparency and facilitating further research.\n\nHowever, while the paper highlights the framework's ability to leverage rich contextual symbols from Source Code Foundation Models (SCFMs), it does not provide detailed examples or case studies illustrating how the basic RAG models contribute to these performance improvements. This can help to understand the contributions of the proposed framework.", "limitations": "See Questions, thanks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Important area.\n+ A novel probe-and-recover framework.\n+ Performance is good.\n+ Open source.", "weaknesses": "- Missing examples of RAG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for your kind feedback! We are delighted to hear that you consider our work in this important area to be both novel and effective.\n\nPlease see our response below.\n\n> ### Q1. Missing example of RAG\n\nWe show two examples of RAG in the uploaded PDF file. Both figures can be interpreted similar to Figure 11 in the submission. The code snippets retrieved by a retriever are shown in yellow blocks. In Figure 1 of the uploaded PDF, we can see that RAG helps LLM generate a more context-relevant summary. That is because the datastore contains code snippets that are very similar to the query function (e.g., the function `sp_256_ecc_recode` in Figure 1c is a crypto related function that performs bitwise operations).\n\nOn the other hand, RAG is less helpful than ProRec when the datastore does not contain functions similar to the query function. For example, in Figure 2 of the uploaded PDF, the query function pops an element from a queue. The datastore does not contain similar functions, so the retriever retrieves two snippets of code that have similar syntactic features (e.g., null pointer checks at the beginning; pointer accesses in the loop condition). The retrieved results are not relevant to the ground truth code context. By contrast, ProRec recognizes local semantic information such as getting an element from a queue, and atomic memory operations. Therefore, the probed code snippets are more relevant to program contexts even if the entire query program is not in the datastore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTpW2TWhtc", "reviewer": "Reviewer_9NVM", "summary": "This paper presents a method for Human-Oriented Binary Reverse Engineering (HOBRE) tasks based on Large Language Models (LLMs). In summary, the authors instruct an LLM to generate the desired answer directly and augment their prompt with the idea of Chain-of-thought and few-shot examples. To get the few-shot examples, the authors build a prober with the encoder-decoder structure, incorporating a structure-aware binary function model as the encoder and a source code language model as a decoder. The prober receives the target disassembled code and samples several related source code snippets. As for the Chain-of-thought, the authors design an Analysis step to aid the LLM.", "questions": "From the first concern, I want to ask:\n- Why can the source code snippets sampled from the prober help the LLM  generate better answers? Please analyse the reasons in detail.\n\nFrom the second concern, we want to ask:\n- Why are the previous works ignored? \n- How does the retrieval work? Why can your retrieval method find relative source code snippets? \n- Why use the retrieval method proposed in this paper? What about the straightforward solution of using CodeBLEU to search for similar disassembled code and use corresponding source code snippets? \n- Why not try building a probe exploiting the LLMs directly, i.e., let the LLM generate probed context?\n\nWe understand that you face the dilemma of designing baselines yourselves. However, to make your experiments even more sound, i.e., your prober can generate more useful examples than the trivial methods, we raise our questions above.", "limitations": "Yes. The authors have openly discuss their limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Significance: HOBRE is an essential topic to discuss, considering the urgency of reusing legacy software. Since the LLMs have learned much of programming, it is promising to explore their potential for HOBRE. \n\n- Quality: The authors have conducted many experiments to improve soundness, e.g., trying various black-box LLMs, proving the correlation between human preference and auto-metrics, and so on. \n\n- Clarity: The charts and pictures illustrate the proposed method and experiment results. The overall structure and narration style make the paper easy to follow.", "weaknesses": "I have two major concerns about this paper:\n\n(1) This paper lacks explanations of what properties of the samples from the prober help the LLM behave better. It is widely accepted that additional examples in the prompts may improve the LLM’s performance. However, according to Table 1 and Table 2, the additional examples from RAG can have negative effects. Why does RAG fail, but yours works? An explanation is needed.\n\n(2) The baseline setup is not sound. First, the comparison between existing HOBRE works is missing, e.g., Ye et al. [1]. Second, the setup of RAG is not clear. Did you compute h_src for all candidate source code snippets and compare them with h_asm in the form of cosine similarities? If so, how do you confirm that the searched source code snippets are relative to the ground truth since your training target is to find the top-1 similar code, but the experiments use top-k similar snippets? Besides, the training target is to match rather than to be relative. Are the two targets equivalent?\n\n[1] Tong Ye, Lingfei Wu, Tengfei Ma, Xuhong Zhang, Yangkai Du, Peiyu Liu, Shouling Ji, and Wenhai Wang. 2023. CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14740–14752, Singapore. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### Q1. Analyze the reason why prober helps\n\nPlease refer to Q2 in global response.\n\n> ### Q2. Why are previous works ignored?\n\nWe will cite and discuss the related work CP-BCS in our paper. We cited supervised methods that train end-to-end binary summarization models, such as BinT5 [1] and HexT5 [2]. However, as shown in a previous study [3], supervised baselines underperform LLMs with regard to generalizability (HexT5 achieves 6.32% METEOR compared to ChatGPT’s 28.13% for binary summarization on a new benchmark). Therefore, in our paper we primarily use zero-shot LLMs as our baselines.\n\nFollowing the reviewer’s suggestion, we collected the results of CP-BCS on our test set during rebuttal. Note that CP-BCS is a supervised model trained for binary function summarization, whereas ProRec does not require any data for summarization. More importantly, their summarization target is the docstring/comment of a function parsed from source code, which is not identical as the summarization targets in our experiments which are LLM summarizations from source code. For a fair comparison, we prepend the comments summarized by CP-BCS to the decompiled code as additional context for LLMs (gpt3.5-turbo-1106)  to revise it into their own summarization styles, so that the final candidate summaries can be properly compared with reference source code summaries. Here,  “+CP-BCS comment” means we augment the decompiled code with the comment for LLM to summarize. If we only evaluate the comments generated by CP-BCS, the CHRF drops to 5.44.\n||CHRF|G4-F|G4-C|\n|-|-|-|-|\n| direct-prompting|30.4|3.6|3.8|\n|+CP-BCS comment|29.0|3.0|2.8|\n|+retrieval|31.7|3.7|3.9|\n|+ProRec|33.5|4.2|4.0|\n\nWe can see that CP-BCS comments have negative impacts on direct-prompting results on our test set, potentially due to the distribution difference between training and test data. Moreover, we cannot easily adapt/transfer CP-BCS to this distribution since the training requires comments within the source code which do not exist in many functions in our training data. It is possible to distill summarization from LLMs, but the cost is high given the large amount of data. For ProRec, data is less of a problem since all the compilable projects can be used to produce binary-source pairs that can be used for alignment.\n\n[1] Al-Kaswan et al. Extending source code pre-trained language models to summarize decompiled binaries. 2023 SANER.\n\n[2] Xiong et al. HexT5: Unified Pre-Training for Stripped Binary Code Information Inference. 2023 ASE.\n\n[3] Shang et al. How far have we gone in stripped binary code understanding using large language models. 2024 arXiv.\n\n> ### Q3. How does the retrieval work? Why can your retrieval method find relevant source code snippets?\n\nAs we discussed in the global response, our retrieval baseline is standard, following the common practice, and is able to retrieve relevant source code snippets if they exist.\n\nWe agree that function-level similarity score might not be equivalent to ideal relevance score for retrieval within the context of HOBRE. However, how to define such ideal relevance is itself a research problem that needs to be further studied. We will try to explore how to define ideal relevance for binary and source code for HOBRE in future work, which might lead to better retrieval methods that are more suitable for HOBRE tasks.\n\n> ### Q4. Why use the retrieval method proposed in this paper? What about the straightforward solution of using CodeBLEU to search for similar disassembled code and use the corresponding source code snippets?\n\nBinary similarity[1-3] (i.e., binary-to-binary search) is a long studied field that we are quite familiar with (these techniques leverages more complex structures such as control-dependence, data-dependence, and dynamic traces, which are more accurate than CodeBLEU score which is based on AST and n-gram). Despite these successes, an assumption that binary similarity techniques can solve HOBRE is that there always exists such “similar code snippets” within the existing knowledge base (collected before hand), which is hardly true in practice as we mentioned above.\n\nIn (easier) use cases where we would actually encounter binaries within the datastore, binary similarity tools can be first leveraged as a filter. However, it’s still necessary to have ProRec for unseen functions.\n\n[1] Pei et al. Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity. 2022 TSE\n\n[2] Wang et al. jTrans: Jump-Aware Transformer for Binary Code Similarity. 2022 ISSTA.\n\n[3] Want et al. CEBin: A Cost-Effective Framework for Large-Scale Binary Code Similarity Detection. 2024 ISSTA\n\n> ### Q5. Why not try building a prober exploiting the LLMs directly, i.e., let the LLM generate probed context?\n\nLeveraging black-box LLMs as probers is challenging because they are not heavily pre-trained on binary code and have limited understanding of it. ProRec addresses this through alignment training.\n\nTo demonstrate this empirically, we conduct experiments on binary function name recovery. We first prompt a black-box LLM (gpt3.5-turbo-1106) to translate decompiled functions into readable ones, sampling multiple results as diverse probed contexts. Using the same prompt as ProRec and the same LLM, we perform function name recovery with additional context. We call this method \"self-probing.\"\n\nThe following table is the performance of self-probing (gpt3.5-turbo-1106) compared to direct-prompting, RAG, and ProRec on 100 randomly sampled test data. (“RoL” stands for ROUGE-L, “P” for precision, “R” for recall, “F” for F1 score).\n\n||SymLM-P|SymLM-R|SymLM-F|RoL-P|RoL-R|RoL-F|\n|-|-|-|-|-|-|-|\n|direct-prompting|16.47|19.40|16.79|53.76|41.07|45.27|\n|+retrieval|17.75|22.05|18.72|58.61|41.30|47.10|\n|+ProRec|20.38|26.72|21.84|58.89|44.54|49.26|\n|+self-probing|16.02|20.52|17.01|54.56|41.86|46.06|\n\nWe can see that self-probing performs slightly better than direct-prompting but is not comparable to RAG or ProRec.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFQZAnbNBE", "reviewer": "Reviewer_CUjj", "summary": "The paper focus on Human-Oriented Binary Reverse Engineering (HOBRE) task. The author propose a probe-and-recover framework that incorporates a binary-source encoder-decoder model and LLMs for binary analysis. The proposed approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments as context. The experiment shows that additional context enables LLMs to enhance recovery accuracy.", "questions": "- What is the purpose of the user study? Is it just for the statement that CHRF is consistent with human preferences in line 216?\n- What is the reason for only adopting ROUGE-L and using the precision, recall, and F1 score for ROUGE-L? Could you present a clearer explanation for the choice of metrics?", "limitations": "The authors need to discuss more on limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ interesting idea\n+ several baselines", "weaknesses": "- hard to follow\n- lack of human evaluation\n- some settings are confusing\n\nThe manuscript offers a promising exploration of the Human-Oriented Binary Reverse Engineering (HOBRE) task. Nonetheless, there are a few areas where the presentation could be enhanced for clarity and impact. For instance, the positioning of Tables 1 and 2 could be revisited to avoid confusion, and the explanation of the metrics discussed on lines 214 and 232 could benefit from further clarification to aid reader comprehension.\n\nGiven the human-centered nature of the task, the inclusion of a user study is highly appropriate. I appreciate the effort to include such a study in Appendix E, though its placement and the clarity of its conclusions might limit its impact. It would be advantageous if some human evaluation, particularly in comparison with baseline methods, could be featured more prominently within the main body of the text.\n\nAdditionally, the selection and justification of evaluation metrics warrant deeper discussion. The introduction of a GPT4-based metric for binary summarization on line 219 is intriguing, yet the absence of a detailed explanation within the main text may leave some readers questioning its validity. Moreover, the decision to exclude commonly used metrics such as BLEU and METEOR, while only including ROUGE-L, could be more thoroughly justified. Providing a comprehensive presentation of all generally employed metrics for summary task and their results would enhance the paper's credibility and thoroughness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### Q1. Presentation\n\nWe will improve the presentation for clarity and impact, such as the positioning of Table 1 and Table 2.\n\n> ### Q2. What is the purpose of the user study? Is it just for the statement that CHRF is consistent with human preference in line 216?\n\nThe user study is a crucial component of the study regarding meaningful metrics for the binary summarization task. Binary summarization is not general text generation, but rather a reverse-engineering task. As such, the metrics that apply to general text generation might not reflect additional properties of generated summary such as **context relevance** and **functionality**.\n\nIn our preliminary study, we noticed that LLMs respond in different styles (e.g., different wording and order) while (1) generating summary directly from decompiled code and (2) generating summary with additional contexts. As our reference summary is generated by an  LLM given source code, the summary style is more similar to that of direct summarization and differs from context-augmented summarization (i.e., RAG and ProRec). Therefore, commonly used metrics can be easily influenced by the style difference and cannot reflect real performance differences on context relevance and functionality which reverse engineers care about.\n\nTo measure the true performance of each approach, we carry out user study and meta-evaluation of commonly used metrics. Metrics for automatic evaluation of general text-generation tasks are widely studied in natural language processing [1, 2, 3]. We follow these works to leverage correlation between automatic metrics and human evaluation as the meta-metric for evaluation. As shown in Figure 5 in our paper, we meta-evaluated BLEU [4], METEOR [6], ROUGE-L [7], and CHRF [8], among which CHRF has the highest correlation with human scores for both context relevance and functionally.\n\nMoreover, recent studies [3,5]  have shown that LLM-as-a-judge becomes more correlated with humans for evaluating text generation in many aspects such as naturalness, coherence, engagingness, and groundedness. Thus, we propose an LLM-based metric for binary summarization. As shown in Figure 5, the correlation of the LLM-based metric has higher correlation with human scores compared to all traditional automatic metrics for both context relevance and functionally.\n\nTo conclude, CHRF and the LLM-based metric we propose are the two metrics that are most consistent with human scores and we report them as our final metrics for evaluating binary summarization.\n\n[1] Zhang et al. Bertscore: Evaluating text generation with bert. 2019 arXiv.\n\n[2] Yuan et al. Bartscore: Evaluating generated text as text generation. 2021 NeurIPS. \n\n[3] Zheng et al. Judging llm-as-a-judge with mt-bench and chatbot arena. 2023 NeurIPS.\n\n[4] Papineni et al. Bleu: a method for automatic evaluation of machine translation. 2002 ACL.\n\n[5] Chan et al. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. 2024 ICLR.\n\n[6] Banerjee et al. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. 2005 ACL Workshop.\n\n[7] Lin et al. Rouge: A package for automatic evaluation of summaries. 2004 Text summarization branches out.\n\n[8] Popović et al. chrF: character n-gram F-score for automatic MT evaluation. 2015 SMT Workshop.\n\n> ### Q3. What is the reason for only adopting ROUGE-L and using the precision, recall, and F1 score for ROUGE-L? Could you present a clearer explanation for the choice of metrics?\n\nWe would like to clarify that  \"ROUGE-L\" for evaluating binary function recovery (Table 2) is essentially at the character level, which indicates a finer granularity than subword-level precision and recall which is used by SymLM [1] metrics. This helps to avoid some limitation of the tokenization. This kind of character-level metrics, such as character-level BLEU, have been widely used in NL2Bash command generation tasks [2, 3]. We will make this clearer in our paper by calling it charROUGE-L.\n\nIn fact, character-level metrics show consistent results with SymLM metrics and char-ROUGE-L. We show more character-level metrics on binary function recovery below.\n\n||charBLEU|charMETEOR|charCHRF|charROUGE-LSum|\n|-|-|-|-|-|\n|direct-prompting|11.94|38.08|17.67|33.88|\n|+retrieval|10.84|38.35|17.40|33.21|\n|+ProRec|14.38|41.10|20.69|36.23|\n\n\nAs function names are typically brief and precise, without much verbose descriptions as in summarization, these statistics-based metrics are all informative. The reason we only show charROUGE-L is that we think that charROUGE-L is more intuitive for binary function recovery, as high charROUGE-L shows a longest common subsequence overlap between the prediction and the reference which means a similar function name that can hint reverse engineers. Overall, we believe the metrics do demonstrate that our method is better than baselines for binary function name recovery.\n\n[1] Jin et al. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. 2022 CCS.\n\n[2] Lin et al. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. 2018 LREC.\n\n[3] Shi et al. Natural Language to Code Translation with Execution. 2022 EMNLP.\n\n[4] Zhou et al. DocPrompting: Generating Code by Retrieving the Docs. 2023 ICLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers for their insightful questions and suggestions! We are glad that the reviewers recognized our paper for studying an "interesting and important problem," being "well written," addressing a "critical and emerging area," presenting a "novel framework," and offering "promising results."

Below we address some common concerns from reviewers.

> ### Q1. The reason why our prober performs better than retriever for context augmentation

Our targeted use scenario is a typical one in practice, in which functionally equivalent binary functions may not exist in the datastore collected beforehand, analogous to zero-day scenarios in cyber-security. Therefore, the task is intrinsically difficult.

In our experiments, we strictly deduplicated the test set from the training set by source functions to ensure we are evaluating the true generalizability of the model on binary summarization and function name recovery.


Our hypothesis is that, the generalizability of models that produce relevant contexts should come from recovering relatively local parts of the binary functions, which might appear in functions from datastore or training corpus.  When local recoveries are properly handled, it becomes possible that HOBRE systems generalize to new functions as unseen compositions of these local recoveries, given the strong ability of LLMs to aggregate information, reason and summarize. Our prober therefore works better than a retriever in three aspects:
- **Fine-grained Representation**: compared to baseline dense retrievers that encodes each binary function into one single feature vector, our prober is better at capturing fine-grained local features by encoding each function into multiple node token embeddings (decided by the architecture and training), thus preserving more local information.
- **Knowledge**: The knowledge possessed by the prober’s SCFM component, considering gigantic pre-training corpus of source code, is far more than that within the limited datastore leveraged by RAG systems, since the latter contains binary-source pairs that require successful compilation and accurate mapping between binary and source functions.
- **Flexibility**: The retrieval ultimately produces source functions as a whole from the datastore, which can hardly be functionally identical to the query binary. Therefore, even if relevant, such source functions will introduce noise. In contrast, the prober synthesizes such contexts, flexibly translating them into human understandable symbols and local structures such as loops (as described in section 2.2). The synthesis is sampled multiple times to mitigate variance. Thus, a well-aligned prober potentially introduces less noise.

> ### Q2. More details about the retriever we use

During inference, our retrieval baseline ranks **all** the source code snippets from the datastore based on the cosine similarity scores between the query embedding h_asm and the embedding of each source code snippet h_src. The top-k source code snippets are used as additional context.

For training, contrastively pre-trained retrievers that use cosine similarity are commonly adopted in most modern dense retrieval systems from OpenAI or Google [2, 3]. In this paper, we follow the common practice to experiment with the most straightforward way of cross-modal retrieval (which is similar to the SOTA in the field [1]) for our retrieval-augmented baseline. Since the SOTA model is not public, and for fair comparison, we trained our own cross-modal retriever with similar contrastive objectives on the same dataset as our prober. Such models can retrieve source code given binary because trained dual-encoders can map binary and source functions to the same embedding space where semantically similar functions’ embeddings are close to each other. Our retriever has a high (84%) recall@1 on its own validation set. The limitation of the retriever on HOBRE is that there might not exist such relevant code snippets within the knowledge base as we previously mentioned.

[1] Jiang et al. BinaryAI: Binary Software Composition Analysis via Intelligent Binary Source Code Matching.  2024 ICSE.

[2] Neelakantan et al. Text and code embeddings by contrastive pre-training. 2022 arXiv.

[3] Ni et al. Large Dual Encoders Are Generalizable Retrievers. 2022 EMNLP.

Meta-review: This paper presents a method for reverse engineering binaries based on LLMs. Inspired by multimodal models in other domains, this work uses a specialized binary (code) encoder to encode structural properties of the binary being reversed. This allows training the "prober" LLM which generates noisy but potentially useful candidate snippets that a powerful recoverer LLM (GPT-x, Gemini, Claude) can use without fine-tuning.

This work is technically sound and contains the interesting novelty of a (binary) code multimodal model that incorporates the structure of the analyzed binary and the idea of using a universal self-consistency/RAG-like idea for combining multiple samples from weaker but more customizable models (the "recoverer").

On the negative side, the evaluated tasks are relatively weak (NL summarization of binaries, and function naming) which reflect a small portion of binary reverse engineering tasks. Importantly, this work does not seem to include adversarial settings (e.g., malware, obfuscated code) that would be more challenging for existing models.

Given the above, I believe that there is some value in publishing this work at its current state as it would be useful for research in the cross-section of LLMs and code.
Decision: Accept (poster)