Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases

Authors: zian  su
xiangzhe  xu
ziyang  huang
kaiyuan  zhang
xiangyu  zhang
purdue  university
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 04:55:28

Abstract: Human-Oriented Binary Reverse Engineering (HOBRE) lies at the intersection of binary and source code, aiming to lift binary code to human-readable content relevant to source code, thereby bridging the binary-source semantic gap. Recent advancements in uni-modal code model pre-training, particularly in generative Source Code Foundation Models (SCFMs) and binary understanding models, have laid the groundwork for transfer learning applicable to HOBRE. However, existing approaches for HOBRE rely heavily on uni-modal models like SCFMs for supervised fine-tuning or general LLMs for prompting, resulting in sub-optimal performance. Inspired by recent progress in large multi-modal models, we propose that it is possible to harness the strengths of uni-modal code models from both sides to bridge the semantic gap effectively. In this paper, we introduce a novel probe-and-recover framework that incorporates a binary-source encoder-decoder model and black-box LLMs for binary analysis. Our approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments as context. This additional context enables black-box LLMs to enhance recovery accuracy. We demonstrate significant improvements in zero-shot binary summarization and binary function name recovery, with a 10.3% relative gain in CHRF and a 16.7% relative gain in a GPT4-based metric for summarization, as well as a 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results highlight the effectiveness of our approach in automating and improving binary code analysis.

Main paper: 
 introduction: 
 In recent years, we see two trends of uni-modal code model pre-training. On one hand, there is a remarkable surge in the development of generative Source Code Foundation Models (SCFMs) [12, 65, 55, 20, 43] , along with advancements in general Large Language Models (LLMs) [4, 60, 46] . Driven by a growing interest in automating software development, these powerful models are trained on billions of tokens from diverse codebases, covering a wide spectrum of programming languages [33] . They possess the capability to complete, infill [19] , and refine code [67] , as well as generate code from natural language instructions [44] . On the other hand, there is a stream of research focusing on binary understanding models [61, 58] , which target learning nuanced code semantics with structures of low-level code, which is critical for software security. Both fields are evolving through continuous pre-training on expansive datasets of uni-modal data, setting new benchmarks in both effectiveness and complexity.Human-Oriented Binary Reverse Engineering (HOBRE), which involves automatically lifting binary to human understandable contents [14, 69] , typically source-code related, occupies a unique Binary FunctionThe purpose of the function is to calculate the cumulative sum of the elements in the input array and store the results in the array , while also updating the values in the array.The potential source function 2 is highly relevant to the decompiled function as it seems to perform a similar operation of updating values in two arrays based on the current iteration count.similar loopsFigure 1 : The ProRec Framework for human-oriented binary reverse engineering. The figure shows a simple example of lifting a cumsum function from binary to human readable summarization. The probed contexts synthesized by the cross-modal knowledge prober, while not identical to the oracle source code of the query binary, exhibit informativeness in terms of symbol names and correct loop structure. These contexts help the black-box LLMs to successfully recover the high-level functionality of binary function in the summary that is consistent with the source code summary, moving beyond merely describing its low-level operations.intersection between these two fields. Existing decompilers for binary reverse engineering are able to translate binary code to C-style code that is functionally equivalent to its original source code. However, a significant semantic gap remains between the decompiled code and its original source, primarily due to the absence of meaningful symbolic information in binary code. Hence, human expertise is still indispensable in the reverse engineering process. HOBRE aims to bridge this semantic gap, which traditionally requires substantial human effort, by leveraging cross-modal deep learning models. Existing approaches either train task-specific small expert models in a supervised manner [30, 1, 69, 75] , which lack generalizability as shown in later evaluations [56] , or require extensive continual pre-training of uni-modal SCFMs [28] which is undesirable considering cost and the risk of forgetting previously acquired source code knowledge [32, 77] . There are also attempts in directly prompting LLMs for HOBRE, which, even though demonstrates better generalizability than small supervised models, also face challenges in understanding stripped decompiled code that lacks symbolic information [29] .Our insight is that this semantic gap between binary and source code is analogous to the gap between low-level pixels in images and high-level concepts in natural language, which can be bridged with sufficient understanding of both. Inspired by the achievements of multi-modal models that seamlessly integrate vision, audio, or other signals with language to facilitate reasoning [2, 37, 42, 45] , we hypothesize that HOBRE could similarly benefit from leveraging uni-modal models developed for both source code and binary code. Such integration would enhance our ability to bridge the semantic gap and enable more effective semantic lifting.In this paper, we validate this idea by proposing a novel probe-and-recover framework ProRec that incorporates a binary-source encoder-decoder model and black-box LLMs for HOBRE, featuring a compute-efficient cross-modal alignment approach of a binary function encoder and a frozen SCFM for the binary-source model. The workflow of ProRec is shown in Figure 1 . The aligned binarysource model acts as a cross-modal-knowledge-prober that can synthesize symbol-rich, diverse source code fragments condition on binary input, denoted as probed contexts. The black-box LLM functions as recoverer that takes as input the binary function together with the probed contexts for tasks such as binary summarization. Intuitively, the conditional source code synthesis by the aligned binary-source code model can be viewed as probing the base SCFM as a parametric knowledge base [52] with a binary function as query, given that the SCFM's weights remains unchanged before and after the alignment. A black-box LLM analyzes and aggregates these knowledgable contexts with the binary function for recovery. This way, ProRec leverages both cross-modal aligned knowledge and strong reasoning ability of LLMs and can outperform directly letting the LLM to reason. ProRec is general and can be applied to different base architectures, continually evolve with base models.We demonstrate the effectiveness of ProRec on two core tasks in reverse engineering [9, 10] : binary summarization and binary function name recovery. The former aims to generate natural language descriptions for a binary function, and the later aims to recover the function name of a decompiled function. We evaluate ProRec on a diversified dataset compiled from GitHub repositories, demonstrating improvements of 3.1% (10.3% relative gain) in CHRF and 12% (16.7% relative gain) in a GPT4-based metric that has high correlation with human judgement on the summarization task over zero-shot baseline. We conduct human study to show the effectiveness of the newly proposed GPT4-based metric. On name recovery tasks, ProRec significantly improves over zero-shot baseline by 6.7% and 7.4% for token-level precision and recall, respectively. For both tasks, ProRec also consistently show advantage over a retrieval-augmented baseline with a strong cross-modal dense retriever. 22 ProRec: Reverse Binary by Probing Source Code Foundation ModelsIn this section, we first present the ProRec framework in §2.1. Next, we describe the neural architecture used for the cross-modal knowledge prober and recoverer in §2.2. The training for the prober in is detailed in §2.3, followed by the comphrehensitve explanation of the knowledge probing stage in §2.4.Formulation Given a binary file, we can leverage binary analysis tools 3 to obtain each binary function x. Specifically, x can either be in its disassembled code form which we denote as x asm , or its stripped decompiled code form, denoted as x dec . x asm and x dec are semantically equivalent and similarly unreadable. The goal is to recover human readable information y given x dec and x asm . 
the probe-and-recover framework: 
 ProRec assumes a binary understanding model parameterized by θ, an open-source SCFM parameterized by ψ, and a black-box LLMs by ϕ. As illustrated in Figure 1 , the binary model together with the SCFM form the cross-modal knowledge prober. The black-box LLM serves as a recoverer. The cross-modal prober can synthesize source code fragments given binary input. The recoverer takes in augmented context with binary code to analyze and perform final recovery.Conceptually, the ProRec framework decomposes the probability to generate y into three parts, the probability of a set of k source code fragments S k = {s 1 , • • • , s k } being relevant to input P (S k |x), the probability of LLM's relevance analysis of the source code fragments P (A|S k , x), and the probability of generating the recovery results conditioned on the analysis and source code fragments.EQUATIONThe decomposition is similar to that of retrieval-augmented generation [36, 68] , where p(y|x) = s∈top-k(S * ) P (y|s, x)P (s|x), given a document pool S * . However, there are two major differences. First, the source code fragments S k are not retrieved from S * , instead, they are sampled from the conditional distribution of the prober P θ,ψ (•|x). Due to the alignment strategy (discussed in §2.3), source code fragments sampled from the prober's distribution have more flexibility than those retrieved a fixed document pool in binary reverse engineering scenario, potentially less noisy. We empirically demonstrate the superiority of probing over retrieval for augmentation in §4.Second, we stress the internal analysis from the LLM denoted as P ϕ (A|S k , x) in the decomposition. The insight is that, even though high-level recovery requires additional domain information to hint black-box LLMs for further induction, that doesn't necessarily mean LLMs totally lacks of such knowledge (since proprietary LLMs can be significantly larger than open-source code language models in size and may have more training data, just different mixtures), it might be some long-tail knowledge that requires better prompting to exploit [35, 76] . On the other hand, the analysis help LLMs to be less influenced by the noisy contexts. This is beneficial for both retrieved and probed contexts.Note that, in Equation 1, the final probability is marginalized over all possible S k and A. In practice, we take the most probable S k and keep the analysis with in the response before final result, without heavy sampling. We will discuss the sampling of each s within S k in §2.4.ProRec is a general framework and is not bounded to existing models and architectures. This section is our current implementation of the prober and recoverer that provide the best performance in our experiments.The core of ProRec is the cross-modal prober, which is an encoder-decoder model aligned in the token embedding space of the SCFM, as illustrated in Figure 2 . We would like to benefit from both pretrained binary function encoders that possesses binary domain knowledge and the strong SCFMs for generalizable probing. We choose the state-of-the-art CODEART [58] as our structure-aware binary function encoder g(•). CODEART is a BERT-like transformer enccoder that takes as input a disassembled binary function x asm along with its dependency graph G dep obtained by program analysis, and outputs the embeddings for all assembly code tokens and graph node tokens (each graph node token corresponds to one instruction, e.g., mov rax, [rbp+var_40], in the assembly code). We choose the Code-Llama [55] family as our base SCFM 4 .For the final prober architecture, we apply a simple two-layer MLP to project the node token embeddings Z asm , with indices N_IDX in all token embeddings, to source code token embeddings space.EQUATIONwhere l n denotes the number of node tokens, d b is the dimension of the binary encoder and d s is the dimension of the SCFM. The projected embeddings are fed into the SCLM as an additional prefix before regular subtoken embeddings for conditional generation.We only use node token embeddings as binary features due to their significantly smaller quantity compared to all token embeddings (approximately one eighth) since assembly functions tend to be long. These embeddings also already capture some structural abstraction of binary code which is meaningful in HOBRE tasks.Recoverer We leverage proprietary black-box LLMs (GPT3.5, Claude-3, and Gemini-Pro) as our recoverer, since they have strong reasoning ability and support long contexts. Specifically, the LLMs are prompted with x dec as zero-shot baseline. For retrieval-augmented baseline and ProRec, we append the additional context to the original input and instruct LLMs to analyze relevance and then generate recovery. Detailed prompts can be found in Appendix D.The training of prober contains two stages: the pre-alignment of the binary encoder to a source code encoder, and the binary encoder-SCFM alignment. Both utilize data in the form of paired binary-source functions. The goal is to gradually align the binary encoder with the base SCFM with minimum knowledge loss.Contrastive Assembly-Source Code Pre-Alignment Since CODEART is exclusively pretrained on binary code corpus [58] , we first align it with a pre-trained source code encoder codet5p-embedding-110m [64] in the function-level embedding space as a pre-alignment stage in order to facilitate the later encoder-decoder alignment. To achieve this, we add a projection head for each encoder to project their [CLS] token embeddings to the same dimension d enc , forming a standard dual-encoder [27] . This dual-encoder can encode (x asm , G dep ) into h asm ∈ R denc and x src into h src ∈ R denc . We train the dual-encoder in a CLIP-like symmetric contrastive fashion [54] . Since the implementation is relatively standard, we refer readers to Appendix A for details.The dual-encoder can function as a dense retriever to score and retrieve the top-k source functions for a query binary function from the source function pool of the training set, based on the similarity measure sim(x asm , x src ) = cos (h asm , h src ). It achieves 84% recall@1 on the validation set with a pool of 10k examples, demonstrating strong performance as a retriever. We utilize this dual-encoder to set up a retrieval-augmented HOBRE baseline to compare with ProRec in §4.Compute-Efficient Cross-Modal Prober Alignment For encoder-decoder alignment, we freeze all the parameters within the SCFM because we intend to explore the extreme end of probing knowledge from it. We freeze CODEART from the first stage except for the last layer which is a transformer block for fast convergence and avoid too much change in the representation. The MLP is fully trainable. The objective of the alignment is to maximizeEQUATIONThe limited amount of trainable parameters results in efficient training. For memory efficiency, we apply quantization (4bit or 8bit) [17, 18] to the base SCFM during alignment.One evidence that the knowledge of the aligned prober is mainly from the SCFM pre-training instead of learned during alignment is shown in Figure 3 . We sampled 500 (x asm , x src ) pairs from the validation set and find that the negative log-likelihood -log P ψ (x src ) for x src provided by the base SCFM and -log P θ,ψ (x src |x asm , G dep ) for x src conditioned on the x asm provided by the aligned prober are highly correlated, indicating that the prober's ability is consistent with the base SCFM. Another interesting observation is that, instruction-tuned SCFMs typically show higher losses during alignment than their original models, which also implies the significance of pre-trained knowledge of source code for cross-modal ability as instruction-tuning may cause forgetting.For the probing process, i.e., sampling S k with the aligned P θ,ψ (•|x asm , G dep ), we want to cover a diverse yet relevant set of candidates. We leverage nucleus sampling [24] to first let the prober generate a relatively large set of source function signatures with high randonmess (top-p = 0.75). We use idea similar to retrieval by training a binary-signature dual-encoder to rank generated signatures and filter out the noisy ones. Ultimately, we use the prober to further complete the remaining signatures with smaller randomness (top-p = 0.5). Since signature is short and important for HOBRE, our strategy achieves both better relevance compared to using a fixed small p for full function generation and better efficiency compared to sampling a large set of functions with a large p. 
experiment setup: 
 We evaluate ProRec on two binary reverse engineering tasks: summarizing function semantics from decompiled code ( §3.1), and recovering function names from decompiled code ( §3.2). In this section, we first introduce our dataset, and the setups of each task.The training and evaluation of ProRec requires pair-wise data between a binary function and its corresponding source code. To the best of our knowledge, there is no publicly available dataset that contains matched source code with the binary program. Therefore, we follow a widely adapted practice in the reverse engineering domain [13, 34, 7] , using GHCC 5 to automatically clone and compile repositories from GitHub. After the compilation, we map the resulting binary programs with the corresponding source code functions leveraging the debug information in binary programs. In total, our data consists of 270k pairs of binary and source code functions. We split 260k data samples for training and 10k data samples for test. We use 5% of the training data as the validation dataset. To make the evaluation cost tractable, we randomly sample 1k samples from the test dataset. For details in data processing and quality assurance, please see Appendix B.1.A binary summarization tool takes as input a snippet of decompiled code, and outputs natural language descriptions. It facilitates two key processes in the reverse engineering practice [9] : understanding the purpose and domain of a program (referred to as context relevance), and understanding the functionality of a program (functionality). Please see Appendix F for a detailed example.Setup We instruct an LLM to summarize decompiled code with three setups: (1) providing the model with only the decompiled code; (2) additionally providing the relevant source code snippets retrieved from the datastore consisting of all source functions in prober's training set by the crossmodal retriever; (3) additionally providing and the source code snippets generated by ProRec. The first two setups are considered baseline approaches for comparison. We further instruct each LLM to summarize the source code corresponding to the test samples as reference summaries.Metrics Given the reverse engineering nature of binary summarization, the automatic evaluation metrics should reflect context relevance and functionality of the summary, different from text summarization. For final results, we report CHRF [53] , which our meta-evaluation (described next) identified as the most aligned with human preferences among popular existing metrics such as BLEU [48] . Additionally, we introduce and report two GPT4-based metrics for context relevance and functionality judgement respectively, following LLM as a Judge [78] , which demonstrate strong correlation with human judgments. The GPT4-based metrics range from 1 (worst) to 5 (best) based on corresponding criteria. Further details (e.g., prompts and rationale) about the GPT4-based metrics can be found in Appendix B.2 and Appendix D.User Study We conduct a user study 6 to gather human judgments on the quality of binary summarization, which serves as the gold standard for this task. The study aims to (1) perform a metaevaluation of automatic metrics and (2) accurately assess the performance of different summarization approaches. Participants are asked to score a summary based on decompiled code, corresponding source code, and the reference summary. The scoring is done on two criteria-context relevance and functionality-on a scale from 1 (worst) to 5 (best). The method used to generate each summary is not disclosed to the participants. For the meta-evaluation of automatic metrics, we calculate the Table 1 : Main results of binary summarization and binary function name recovery. "G4-F" and "G4-C" denote GPT4Evaluator for functionality and context relevance, respectively. P SymLM , R SymLM , F SymLM denote token-level precision, recall, F-1 score as in SymLM [30] . "cBLEU", "cRoL", and "cMETEOR" stands for character-level BLEU, ROUGE-L, and METEOR scores. Spearman correlation between human judgments and automatic metric scores. For more details, we refer readers to Appendix E.Different from generating summary for a decompiled function, recovering function name requires more accurate understanding about program contexts and more concise abstraction for program semantics. This assists reverse engineers in efficiently navigating numerous functions in a real world binary program. A detailed example is provided in Appendix F.Setup We use the source code function names as ground truth for name recovery. Similar to the binary summarization task, we conduct experiments with three setups: prompting recoverers with only the decompiled code, with decompiled code and source code snippets obtained by a retriever, with decompiled code and source code snippets generated by ProRec. The first two setups are considered baselines for comparison.Metrics We evaluate the performance of a tool for the binary function name recovery task at different levels of granularity.Token-level Metrics. In line with existing work in reverse engineering [30] , we tokenize both the predicted function name and the corresponding ground truth, then compute precision, recall, and F1 score at the token level. For each metric, we first calculate the scores for individual function name predictions and then average them across all functions.Character-level Metrics. We adapt BLEU [48] , METEOR [8] , ROUGE-L [39] for the function name by tokenizing function names into characters and computing these metrics on character level, similar to [40, 57] . They provide a fine-grained evaluation of the function names and can avoid some limitations of tokenization.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper contains no theoretical result. The approach is evaluated empirically.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
results: 
 In all the following experiments, we report ProRec results based on CodeLlama-34b (4bit quantized).For both the retrieval-augmented baseline (+retrieval) and ProRec (+ProRec), we use their top-5 contexts as augmentation. The versions of the black-box LLM recoverers are gpt-3.5-turbo-1106 for GPT3.5-turbo, claude-3-haiku-20240307 for Claude-3, gemini-1.0-pro for Gemini-Pro, and gpt-4-turbo-2024-04-09 for GPT4 Evaluator. We show the results for binary summarization in Table 1 . Observe that ProRec helps all models generate better summary in terms of CHRF. A retriever, on the other hand, may introduce noise (irrelevant source functions) to a model and even makes the results worse (e.g., for the Gemini-Pro model). Moreover, we can see that ProRec achieves higher scores when evaluated with the GPT4Evaluator on functionality (G4-F) and context relevance (G4-C), indicating the summary of ProRec is more helpful to a human reverse engineer.We further analyze the results of GPT4Evaluator to illustrate the advantage of ProRec. The results for summaries generated by GPT-3.5 are visualized in Figure 4 . It is worth-noting that we define the score 3 as a "neutral" score, meaning that a summary does not contain specific context (for the context relevance question) or contains only correct but low-level operations without high-level abstractions (for functionality question). We can see that for most cases, GPT-3.5 achieves a score with at least 3. That indicates the LLM can largely understand the low-level behaviors of decompiled code. That is because decompiled code is in the C-syntax. On the other hand, we can see that for the context relevance question, both retrieval-augmented baseline and ProRec introduces more useful context information to the model, and thus the resulting summaries have closer relevance to the ground truth source code. Especially, queries with the code snippets generated by ProRec achieve more scores 4 and 5 than queries enhanced with a retriever's results. That illustrates ProRec indeed generates code snippets with better context relevance than a retriever.For the functionality question, we can observe similar patterns. That indicates better contexts introduced by ProRec help the LLM to understand code functionality. We show a detailed example in Appendix F.Human Evaluation Table 2 presents the human evaluation results from our user study for 50 randomly sampled summaries of each method (using GPT-3.5-turbo as the recoverer). The human evaluation aligns with the automatic metrics: ProRec consistently outperforms the other approaches in terms of both context relevance and functionality, according to human judgment.We show results for binary function name recovery in Table 1 . We can see that the code snippets generated by ProRec helps all three LLMs predict better names in terms of token-level precision, recall, and F-1 score designed for reverse engineering task [30] . Especially, ProRec outperforms a retriever by a large margin, indicating that ProRec generates more relevant code than a retriever. For character-level metrics, ProRec shows similar improvements, not biasing towards certain metrics. Note that different LLM recoverers can have different performance on the two tasks, e.g., Gemini-Pro is better at binary function name recovery than summarization compared to other two models, yet this does not influence the improvement ProRec brings to both tasks and all recoverers. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide all the information needed to reproduce the main experiment results. 
analysis: 
 How does black-box LLM's internal analysis A help robust recovery? We study the influence of LLM's internal analysis by evaluating retrieval-augmented recovery and ProRec with different number of additional contexts, since we believe this kind of internal analysis is crucial for LLM-based recoverers to perform robust binary recovery with additional contexts, especially when the provided contexts are noisy. We run binary function name recovery for 100 randomly sampled examples, 3 times, for zero-shot recovery (dec-only), retrieval-augmented recovery (recovery), and ProRec. As shown in Figure 5 , the internal analysis consistently reduce the variance of function name recovery performance of both retrieval-augmented recovery and ProRec. This is particularly true for retrieval when k gets large. We deem that it may due to a lack of function-level similar source code in the data store. On the other hand, we observe sometimes LLM tend to be too conservative without leveraging the extra contexts with the internal analysis, potentially because of our not specifically optimized prompts which can be fixed by making some adjustments. Moreover, we argue that misleading is worse than not informative, and reverse engineers can further interact with LLMs for more aggressive induction after obtaining a conservative response.Ablation Study on Base Source Code Foundation Model Size ProRec's performance relies on the ability of the base SCFM, where size is a crucial indicator since knowledge is represented by model parameters. Therefore, we study the influence of base SCFM size. We train three probers based on CodeLlama-7b, CodeLlama-13b, and CodeLlama-34b, all in 4bit quantization for fair comparison.We report statistics of these three probers in Table 3 . As shown in the table, with growing number of base model size, the prober achieve a lower loss on validation set, which leads to an increase in average n-gram overlap of probed source code fragments and the oracle source function, which we run 3 times on 100 examples for each row. However, n-gram overlap with oracle source function seems not to significantly influence downstream task performance like CHRF for binary summarization. We hypothesize that this is potentially due to the tasks like binary summarization is not very sensitive to subtle symbolic difference, which means we can leverage modest size SCFM for probing instead of large ones, being economic in real practice.Case Study We examine three specific cases to illustrate the performance and limitation of ProRec in Appendix F.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our code and data are available at https://github.com/ziansu/prorec. 
related work: 
 Large Multimodal Models Recent advancements in vision-language models have demonstrated their efficacy across a range of practical applications such as image captioning [70] , visual question answering [5, 16, 3] , and image-text matching [38] . While the limited availability of datasets that align different modalities was perceived as a major impediment to scalability, recent works leverage the knowledge embedded within pre-trained large language models [2, 6, 37, 15, 41, 80] . Beyond their capacity to interpret diverse information modalities such as images [63] and audio [25] , LLMs have increasingly been aligned with graph structures [11, 59] and gained widespread attention. In particular, there have been successful attempts that leverage LLMs for graph data involves the Graph2Text strategy, which transforms graph data into textual representations. This technique has been effectively utilized in several studies [62, 21, 74] . The designs in the prober of ProRec share some similarity with recent LMMs, with a modality-specific encoder, and a SCFM decoder. However, we tackle the binary-source code multi-modality which is largely unexplored compared to popular modalities. Also, the multi-modal prober is used in a larger probe-and-recover framework instead of end-to-end training.Retrieval-Augmented Generation Retrieval-augmented generation is widely applied in knowledgeintensive scenarios, such as question answering [22, 26, 31] , molecule generation [66] , and source code generation [79] . By leveraging a non-parametric datastore, retrieval-augmented approaches decompose knowledge and LMs, can complement some long-tail knowledge or keep the knowledge up-to-date without heavy tuning the model which is costly. ProRec, on the other head, tries to exploit knowledge within a parametric SCLM for black-box LLM-based binary recovery. A closely related work is GENREAD [76] that prompts InstructGPT to generate context instead of retrieval for knowledge intensive tasks. ProRec differs from this work in that binary recovery requires crossmodal understanding and informative contexts cannot be obtained by directly prompting LLMs We introduce specially designed cross-modal alignment to allow informative context generation.Binary Reverse Engineering Advances in machine learning models have been widely used to solve challenging tasks in binary program analysis [51, 49, 61, 71, 58, 50] . However, most work focuses on reasoning binary program, and is not human-oriented. Another stream of work trained smaller endto-end models for individual human oriented tasks, such as variable name prediction [47, 13, 72, 73] , and function name recovery [30] . Nonetheless, these models are not benefiting from pretraining efforts and thus have sub-optimal performance [56] . Preliminary study shows HOBRE remains a challenge for state-of-the-art LLMs [29, 56] . Our efforts attempt to address this challenge, leveraging pretraining knowledge of SCFMs to help HOBRE.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the detailed experimental setups.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we introduced a novel probe-and-recover framework, ProRec, designed to bridge the semantic gap between binary code and human-understandable source code. By integrating an aligned binary-source encoder-decoder model with black-box large language models, our approach effectively synthesizes symbol-rich code fragments from binary input, providing valuable context for improving binary analysis tasks. Our extensive evaluations demonstrate that ProRec significantly enhances performance in both binary summarization and binary function name recovery tasks.Limitations & Future Work We experiment with a simple achitecture and straightforward alignment of the binary-source prober in this paper, which might not be optimal for ProRec. Future work can explore better prober architecture and alignment objectives. Moreover, currently we only focus on intra-procedure analysis, similar to most existing work. In practice, HOBRE needs to deal with full binary with multiple functions. An important direction will be extending ProRec to inter-procedure scenarios, where additional information from the whole program such as call-graph can be leveraged, building program-level binary reverse engineering agents.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We draw confidence intervals in our plots, report p-values for user study. All other experiments are conducted on a relatively large dataset.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We use LLMs that are public available. Our training is conducted using 4 NVIDIA A100s.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have read the code of ethics and confirmed that we do not violate the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: No. Our work focuses on a specific domain of reverse engineering.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We discuss details of the contrastive training of the assembly-source code dual-encoder in this section. Given a mini-batch B = {(x a i , x s i )} N i=1 with batch size N , where x a i represents the i-th tuple of assembly code and its dependency graph, x s i represents the corresponding i-th source code, we train the dual-encoder, g(•) for binary encoder, and h(•) for source encoder, with the following objectiveEQUATIONwhereEQUATIONandEQUATIONHere, we use cosine similarity for sim(•, •). In order to have more negative samples, we use momentum encoders [23] for both modality with a queue size 4096 and momentum 0.999. We train the model with learning rate 5e-5, a batch size of 16, 1k warmup steps, and 17k total steps.The aligned dual-encoder as a cross-modal dense retriever achieves 84% recall@1 on the validation set with pool size 10k, which demonstrates that it has reasonable retrieval performance.We ensure data quality by (1) selecting projects with no less than 20 stars, (2) including only executable binary programs that can be fully stripped, and (3) deduplicating our dataset by checking the source code string.Initially, we obtained 18k projects from Github. We tried to compile all of them in x86-64 with O0, and discarded not compilable ones. It generates 106k executable binaries. We then match binary functions with source code functions by their function names, and deduplicate data samples by their source code strings (e.g., some utility functions may be used in multiple programs). Our final dataset containing 270k pairs of binary and source code functions.We provide GPT4 with the decompiled code, the corresponding source code, and the reference summary to evaluate. For each question, we adapt the Likert scale 7 and instruct GPT4 to output a score from 1 (worst) to 5 (best). The users in our human study are provided with similar information as questionnaires to perform human judgment of summaries.We derive our evaluator prompts from a thorough survey on reverse engineer [9] . The survey summarizes 8 sub-goals of human reverse engineers. We list them in Table 4 and categorize the goals into four scopes. We highlight ones that binary summarization can help.Specifically, for goal (7) , a reverse engineer aims to reason about the high-level abstraction of the program, e.g., what the program does, and how the program works [9] . We use the prompt in Figure 6 to evaluate how helpful a summary is to obtain the high-level picture of a program. • For 5, the summary and the reference should describe the same domain/context.• For 4, the domain of the summary and the reference should be similar and relevant, although may not be exactly the same. The summary domain may be a superset or subset of the reference. The summary domain may be closely related to the reference domain. The summary and reference may be two different perspectives of a same specific domain.• For 3, the summary does not explicitly mention a specific context. It only contains low level operations. From the summary, one cannot deduce the high-level purpose of the decompiled function.• For 2, the summary is slightly misleading. The summary domain is different and not relevant to the reference domain. However, it is implied by the choice of words in the summary, and is not explicitly mentioned.• For 1, the summary is completely misleading. The summary domain is irrelevant to the reference domain, and it is explicitly mentioned in the summary.Your output should first briefly comment the summary from the aforementioned perspectives. Do not allow the length of the responses to influence your evaluation. Be as objective as possible.Figure 6 : Prompts for GPT4-Evaluator for asking context relevance.For goal (8) , a reverse engineer reasons specific behavior individual functions to form the mental models [9] of the program logic. We use the prompt in Figure 7 to illustrate how accurate a summary is to describe the functionality of a program.For other goals in Table 4 , goals (1-2) are associated with specific analyses, instead of programs. Goal (3) aims to capture the general properties of a program (e.g., the size of a program, the sections in a binary executable file). These properties are easily accessible. The following three goals (4-6) are achieved by a decompiler. The decompiler recovers call to the system APIs (goal 4), reasons instructions and lifts them to a C-like syntax (goal 5), and recovers data dependences by introducing variables (goal 6). Therefore, the focus of binary summarization is on the last two goals, requiring understanding and reasoning of program semantics.Function name recovery is important to the reverse engineering task because a human typically starts the reverse engineering task by achieving a rough understanding about all functions, as suggested by studies on human reverse engineers [9, 10] . For example, a malware sample to analyze may contain hundreds of binary functions. A reverse engineer will need to first locate functions with suspicious B. Does the summary reflect relevant functionality? Answer the question in range 5(best) to 1(worst). Functionality means the specific high-level behaviors performed in a function (e.g., sort, string comparison, decoding package, printing error messages).• For 5, the functionality in the summary should be almost exactly the same to the reference.• For 4, the functionalities in the summary are similar to the reference. It may be vague in details, but the overall functionality and purpose is correct.• For 3, the summary does not specify functionality. It only repeats some low-level operations without high level abstractions.• For 2, the summary specify relevant but inaccurate functionality. The functionality specified in the summary may be relevant to the reference summary, but they have significant differences.• For 1, the summary contains irrelevant functionality. It is contains a totally different behavior with the reference.Your output should first briefly comment the summary from the aforementioned perspectives. Do not allow the length of the responses to influence your evaluation. Be as objective as possible.Figure 7 : Prompts for GPT4-Evaluator for asking functionality.behaviors (e.g., executing commands received from a remote server) before analyzing the function in detail. The workload would be huge even if all functions have natural language summaries. On the other hand, if all the decompiled functions have names as in the source code, a human developer can efficiently go through the list of function names and identify functions requiring further inspection.Formally, the token-level precision (P ), recall (R), and F1 are defined as follows:P (i) = T (i) g ∩ T (i) p T (i) p R(i) = T (i) g ∩ T (i) p T (i) g F 1(i) = 2 × P (i) × R(i) P (i) + R(i) ,whereT (i)g is the token set of the ground truth name for the i-th test case, andT (i)p the token set of the i-th predicted name.The precision, recall, and F1 scores for the entire test set are the average scores of individual scores across all test cases. Formally,P = 1 N N i=1 P (i) R = 1 N N i=1 R(i) F 1 = 1 N N i=1 F 1(i),where N is the number of test cases.Different from existing work CP-BCS [75] that supervisedly train a model for binary function summarization, ProRec does not require any superivsed training data for summarization and directly rely on LLM's summarization ability. More importantly, CP-BCS's summarization target is the docstring/comment of a function parsed from source code, which is not identical as the summarization targets in our experiments which are LLM summarizations from source code. For a fair comparison, we prepend the comments summarized by CP-BCS to the decompiled code as additional context for LLMs (gpt3.5-turbo-1106) to revise it into their own summarization styles, so that the final candidate summaries can be properly compared with reference source code summaries. Here, "+CP-BCS comment" means we augment the decompiled code with the comment for LLM to summarize. If we only evaluate the comments generated by CP-BCS, the CHRF drops to 5.44.System: You are an experienced C/C++ software developer. User: You are provided with the following function: {} First generate a brief step-by-step description of its functionality in the format: Description: ... Then generate a high-level summary of its functionality in the format: Summary: The function ... After that, generate a brief description of its general purpose in the format: Purpose: The purpose of the function is to ... To demonstrate this empirically, we conduct experiments on binary function name recovery. We first prompt a black-box LLM (gpt3.5-turbo-1106) to translate decompiled functions into readable ones, sampling multiple results as diverse probed contexts.Using the same prompt as ProRec and the same LLM, we perform function name recovery with additional context. We call this method "self-probing".Table 6 shows the performance of self-probing (based on gpt3.5-turbo-1106) compared to other methods on 100 randomly sampled test data. We can see that self-probing performs slightly better than directprompting but is not comparable to retrieval-augmented recovery or ProRec.We show our prompts to generate source code summarization in Figure 8 , the prompts to generate decompiled code summarization in Figure 10 , and the prompts to recovery function names in Figure 10 . Note that the prompts for GPT4Evaluator are discussed in the previous section, shown in Figure 6 and Figure 7 .The user study involved 12 participants. The participants are PhDs / PhD students that either have some background in reverse engineering or are experienced in C/C++/Rust programming. We ensured each summary is scored by at least 3 users, and use the median scores as the results. The questions in System: You are an experienced binary reverse engineer to understand decompiled C code that lacks symbol information.You are provided with the following decompiled function that is hardly human readable: {} First generate a brief step-by-step description of its functionality in the format: Description: ... Then try to generate a summary of it that can help human understand / inspect its original high-level source code functionality in the format: Summary: The function ... After that, inspect and generate a brief description of its general purpose in the format: Purpose: The purpose of the function seems to ...You are provided with the following decompiled function that is not human readable: {} First generate a brief step-by-step description of the functionality of the decompiled code in the format: Description: ... Then try to generate a summary of it that can help human understand / inspect its original high-level source code functionality in the format: Summary: The function ... After that, consider the following source functions (if any) that are potentially relevant to this decompiled function.{source functions} Analyze whether they are relevant to the decompiled function in the format: Analysis: ... Finally, based on the analysis, try to inspect and generate the general purpose of the decompiled function in the format: Purpose: The purpose of the function seems to ... System: You are an experienced binary reverse engineer to understand decompiled C code that lacks symbol information.You have decompiled a function from an executable, which currently has a generic name like sub_xxx.The decompiled function code is as follows: {} Generate a more human-understandable function name for the decompiled code to replace the original sub_xxx in the format: Function Name: function_name_goes_here User (Augmented):You have decompiled a function from an executable, which currently has a generic name like sub_xxx.The decompiled function code is as follows: {} Consider the following source functions (if any) that are potentially relevant to this decompiled function.{source functions} Analyze whether these source functions are relevant to the decompiled function in the format: Analysis: ... Then, based on the analysis, generate a more human-understandable function name for the decompiled code to replace the original sub_xxx in the format: Function Name: function_name_goes_here our user study were sampled from summaries generated by all three techniques (i.e., ProRec, the retrieval-augmented baseline, and direct prompting baseline).The study leverages the same prompts used in the GPT4Evaluator (details in Appendix B.2) as the questionnaires, asking users to evaluate a piece of summary in terms of context relevance and functionality. As we already show the results of human judgment of different approaches in the main text, here we only show the results of Spearman correlation between human judgment and automatic metrics in Table 7 . We can see that both CHRF and GPT4Evaluator are consistent with human preference, with GPT4Evaluator the most consistent metric with regard to human scores. Therefore, we use CHRF and GPT4Evaluator to evaluate the quality of binary summarizations. In Figure 11 , we show a function that initializes a key for encryption. Without any context information, GPT-3.5 summarizes the function with generic descriptions (e.g., "maniplate and transform the input data"). On the other hand, ProRec generates code snippets (only one of them is shown here) related to encryption keys. Provided with these code snippets, GPT-3.5 correctly summarizes the function as "perform cryptography operations", and mentions several operations related to "key". Although the summarization does not perfectly reflect the "initialization" purpose of this function, the description is clearer and more relevant to the context (i.e., key operations). We can see that retrieval results helps LLM generate a more context-relevant yet not functionally correct summary. That is because the datastore contains code snippets that are very similar to the query function (e.g., the function sp_256_ecc_recode in Figure 11-(d ) is a crypto-related function that performs bit-wise operations). However, the retrieved function does not explicitly mention anything like "key", which prevents the LLM recoverer to further guess the purpose of the function.Figure 12 shows a more extreme case when RAG is less helpful than ProRec with no relevant function similar to the query function in the datastore. The query function pops an element from a queue. The retriever retrieves two snippets of code that have similar syntactic features (e.g., null pointer checks at the beginning; pointer accesses in the loop condition). By contrast, ProRec recognizes local semantic information such as getting an element from a queue, and atomic memory operations. Therefore, the probed code snippets are more relevant to program contexts even if the entire query function is not in the datastore.We study a failure case of ProRec for the function name recovery task. The example is shown in Figure 13 . The function reads data from a temperature sensor and convert the temperature from raw sensor data to human-readable temperature unit. We show two code snippets generated by ProRec.We can see that ProRec successfully recognizes the function is relevant to read and parse data from a sensor. However, it does not accurately associate it with the temperature sensor. Therefore, although the generated summary is of better quality, the recovered function name is still different from the source code.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Our code and data are available at https://github.com/ziansu/prorec.https://hex-rays.com/ida-pro/We also tried other SCFMs like DeepSeek-Coder[20] or StarCoder2[43] in our preliminary study and find that Code-Llama performs best as a base SCFM for our prober.https://github.com/huzecong/ghccWe obtained an IRB for the study (IRB-2024-799).https://en.wikipedia.org/wiki/Likert_scale 


Decision: This paper presents a method for reverse engineering binaries based on LLMs. Inspired by multimodal models in other domains, this work uses a specialized binary (code) encoder to encode structural properties of the binary being reversed. This allows training the "prober" LLM which generates noisy but potentially useful candidate snippets that a powerful recoverer LLM (GPT-x, Gemini, Claude) can use without fine-tuning.

This work is technically sound and contains the interesting novelty of a (binary) code multimodal model that incorporates the structure of the analyzed binary and the idea of using a universal self-consistency/RAG-like idea for combining multiple samples from weaker but more customizable models (the "recoverer").

On the negative side, the evaluated tasks are relatively weak (NL summarization of binaries, and function naming) which reflect a small portion of binary reverse engineering tasks. Importantly, this work does not seem to include adversarial settings (e.g., malware, obfuscated code) that would be more challenging for existing models.

Given the above, I believe that there is some value in publishing this work at its current state as it would be useful for research in the cross-section of LLMs and code.

Reviews: review: paper_summary: paper_summary: This paper presents a new framework, using an encoder-decoder architecture, call ProRec and an LLM black-box model for helping convert binary code in human readable format. The authors try multiple models in an effort to develop ProRec, and settle on using CODEART and Codellama. For black-box LLM they experiment with state-of-the-art models like GPT3.5, Claude-3 and Gemini-Pro.
They generate their own train and test data from GitHub and test on two tasks, summarization and function name recovery. Results improve significantly by using ProRec.
strength_weakness: strength_weakness: The only major weakness that I could see were in-terms of impact of the given approach. 

1. Applicability to other domains:
- The proposed architecture for generating human readable forms of binary code seems very customized for the problem. It is not clear how the given approach can be useful for any other domain apart from the one mentioned.

2. Applicability to other tasks in the domain:
- Since I am not familiar with this particular domain, it is not even clear to me which other tasks beyond function name generation and summarization can benefit from this approach. If the authors feel they can extend the work with more tasks, they should mention it in the paper.
questions: questions: - Which other fields could benefit from the ProRec architecture you propose?

- How do you plan to extend this work in the future?
limitations: limitations: There is no limitations section in the paper. However since they deal with the security domain, they should consider how it may be used by malicious actors.
Is it possible that the work is used by malicious agents to understand better and exploit critical software infrastructure? 

Another limitation that could be addressed is the impact of the work. To me it seems limited in terms of application to other domains.
paper_summary: paper_summary: Human-Oriented Binary Reverse Engineering (HOBRE) seeks to transform binary code into human-readable content that aligns closely with its original source code, effectively bridging the semantic gap between binary and source. While recent advancements in uni-modal code models, including generative Source Code Foundation Models (SCFMs) and binary understanding models, have been promising, their application in HOBRE has been limited by reliance on either supervised fine-tuning or general large language model prompting. This has often led to suboptimal outcomes. Drawing inspiration from the success of multi-modal models, the authors propose a novel "probe-and-recover" framework that synergistically combines binary-source encoder-decoder models with black-box LLMs for enhanced binary analysis. This framework uses pre-trained SCFMs to generate symbol-rich code fragments as context, improving the interpretive capabilities of black-box LLMs (termed "recoverers") used in the process. The proposed approach has demonstrated significant enhancements in zero-shot binary summarization and binary function name recovery tasks. Notably, it achieved a 10.3% relative improvement in CHRF and a 16.7% relative gain in a GPT-4-based metric for summarization, along with a 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results underscore the effectiveness of the authors' framework in automating and refining the process of binary code analysis.
strength_weakness: strength_weakness: - Missing examples of RAG.
questions: questions: The paper explores a critical and emerging area within cybersecurity and software engineering, introducing a novel "probe-and-recover" framework that effectively enhances binary reverse engineering. It demonstrates marked performance improvements in binary summarization and name recovery, achieving significant gains in established metrics. Additionally, the authors have commendably made the code available as open source, promoting transparency and facilitating further research.

However, while the paper highlights the framework's ability to leverage rich contextual symbols from Source Code Foundation Models (SCFMs), it does not provide detailed examples or case studies illustrating how the basic RAG models contribute to these performance improvements. This can help to understand the contributions of the proposed framework.
limitations: limitations: See Questions, thanks.
paper_summary: paper_summary: This paper presents a method for Human-Oriented Binary Reverse Engineering (HOBRE) tasks based on Large Language Models (LLMs). In summary, the authors instruct an LLM to generate the desired answer directly and augment their prompt with the idea of Chain-of-thought and few-shot examples. To get the few-shot examples, the authors build a prober with the encoder-decoder structure, incorporating a structure-aware binary function model as the encoder and a source code language model as a decoder. The prober receives the target disassembled code and samples several related source code snippets. As for the Chain-of-thought, the authors design an Analysis step to aid the LLM.
strength_weakness: strength_weakness: I have two major concerns about this paper:

(1) This paper lacks explanations of what properties of the samples from the prober help the LLM behave better. It is widely accepted that additional examples in the prompts may improve the LLM’s performance. However, according to Table 1 and Table 2, the additional examples from RAG can have negative effects. Why does RAG fail, but yours works? An explanation is needed.

(2) The baseline setup is not sound. First, the comparison between existing HOBRE works is missing, e.g., Ye et al. [1]. Second, the setup of RAG is not clear. Did you compute h_src for all candidate source code snippets and compare them with h_asm in the form of cosine similarities? If so, how do you confirm that the searched source code snippets are relative to the ground truth since your training target is to find the top-1 similar code, but the experiments use top-k similar snippets? Besides, the training target is to match rather than to be relative. Are the two targets equivalent?

[1] Tong Ye, Lingfei Wu, Tengfei Ma, Xuhong Zhang, Yangkai Du, Peiyu Liu, Shouling Ji, and Wenhai Wang. 2023. CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14740–14752, Singapore. Association for Computational Linguistics.
questions: questions: From the first concern, I want to ask:
- Why can the source code snippets sampled from the prober help the LLM  generate better answers? Please analyse the reasons in detail.

From the second concern, we want to ask:
- Why are the previous works ignored? 
- How does the retrieval work? Why can your retrieval method find relative source code snippets? 
- Why use the retrieval method proposed in this paper? What about the straightforward solution of using CodeBLEU to search for similar disassembled code and use corresponding source code snippets? 
- Why not try building a probe exploiting the LLMs directly, i.e., let the LLM generate probed context?

We understand that you face the dilemma of designing baselines yourselves. However, to make your experiments even more sound, i.e., your prober can generate more useful examples than the trivial methods, we raise our questions above.
limitations: limitations: Yes. The authors have openly discuss their limitations.
paper_summary: paper_summary: The paper focus on Human-Oriented Binary Reverse Engineering (HOBRE) task. The author propose a probe-and-recover framework that incorporates a binary-source encoder-decoder model and LLMs for binary analysis. The proposed approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments as context. The experiment shows that additional context enables LLMs to enhance recovery accuracy.
strength_weakness: strength_weakness: - hard to follow
- lack of human evaluation
- some settings are confusing

The manuscript offers a promising exploration of the Human-Oriented Binary Reverse Engineering (HOBRE) task. Nonetheless, there are a few areas where the presentation could be enhanced for clarity and impact. For instance, the positioning of Tables 1 and 2 could be revisited to avoid confusion, and the explanation of the metrics discussed on lines 214 and 232 could benefit from further clarification to aid reader comprehension.

Given the human-centered nature of the task, the inclusion of a user study is highly appropriate. I appreciate the effort to include such a study in Appendix E, though its placement and the clarity of its conclusions might limit its impact. It would be advantageous if some human evaluation, particularly in comparison with baseline methods, could be featured more prominently within the main body of the text.

Additionally, the selection and justification of evaluation metrics warrant deeper discussion. The introduction of a GPT4-based metric for binary summarization on line 219 is intriguing, yet the absence of a detailed explanation within the main text may leave some readers questioning its validity. Moreover, the decision to exclude commonly used metrics such as BLEU and METEOR, while only including ROUGE-L, could be more thoroughly justified. Providing a comprehensive presentation of all generally employed metrics for summary task and their results would enhance the paper's credibility and thoroughness.
questions: questions: - What is the purpose of the user study? Is it just for the statement that CHRF is consistent with human preferences in line 216?
- What is the reason for only adopting ROUGE-L and using the precision, recall, and F1 score for ROUGE-L? Could you present a clearer explanation for the choice of metrics?
limitations: limitations: The authors need to discuss more on limitations.

Comments: title: Score update
comment: We are delighted to see that our responses addressed your concerns. We are vey grateful for your continuous help. We will include all the suggested changes.
Thank you for your clarification and the significant effort you have put into the rebuttal. I am generally satisfied with the responses provided to the concerns raised.

I hope our discussion will help you present your experiments more comprehensively and clearly in the final manuscript. Such clarity will undoubtedly enhance the overall impact and accessibility of your research.

I will not oppose the paper's acceptance, but my final decision will consider the revisions made in response to our discussions and the consensus among the other reviewers.
Dear Reviewer 9NVM,

Thank you for your thoughtful feedback on our paper! We have made every effort to address your concerns and questions. As the reviewer-author discussion period is coming to a close, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns.

We are looking forward to your reply. Thank you once again!

Best regards,

The Authors
>  ### how you can make the probe more Knowledgeable and Flexible without introducing any new noise

If we understood the question correctly, it seems to be mainly about why prober can introduce *less* noise compared to a retriever for context augmentation while being more knowledgeable and flexible. Note that we are not trying to claim that the prober does not introduce “any new noise”. Given that HOBRE tasks are analogous to zero-day scenarios in cybersecurity (details in [global response](https://openreview.net/forum?id=qPpVDzPhSL&noteId=w0qwOq1qg3)), it is always possible that any retriever or prober will provide source code snippets that are not equivalent to the oracle source code, which will introduce certain noise to the augmented context for black-box LLM recoverers.

However, prober introduces less noise than RAG because it leverages source code foundation models as the knowledge base and performs generation instead of retrieval (as discussed in the [global response](https://openreview.net/forum?id=qPpVDzPhSL&noteId=w0qwOq1qg3)). For dense retrievers that retrieve top-k functions from the datastore, noise means completely irrelevant source functions selected as “potentially relevant context” when no source function similar to oracle exists in the datastore. Such noisy context can dramatically influence recoverers’ understanding of the binary function.

On the other hand, the essence of using a prober is to remove the reliance on the (limited) datastore. Instead, it leverages the much larger parametric knowledge base which is the source code foundation model. Specifically, by synthesis, the prober may generate code snippets that better align with the given decompiled code, especially when the datastore does not have code snippets directly related to the decompiled code.
We are very grateful for the tremendous time the reviewer has spent on helping us!

> ### Presentation

We think the suggested presentation is better than what we proposed in the previous response. We will revise following the suggestions.

> ### Possibility of user study data leakage

We did not disclose corresponding techniques to users, so that they did not know which technique is used to generate a summary after the user study. Therefore, we believe there was no data leakage.

> ### User study templates

We checked all suggested studies [1-3 (index of reference provided by the reviewer)]. The study in [1] aims to identify the *keywords* in a snippet of source code that should be included in the code summary, which is different from our task. Therefore, the following discussion focuses on comparing our study with [2] and [3].

The user studies in [2, 3] aim to evaluate source code function summary.  Their  studies are similar to ours, in terms of how to conduct them, participants (three Ph.D students and three senior researchers), and scale  (100 code snippets). In particular, their templates include the following: “For each code snippet, we show the participants the oracle comment and the results from four approaches”; “To ensure fairness, the participants are not aware of where the comments are generated from”; “Each participant is asked to rate … (1) Naturalness … (2) Adequacy … (3) Usefulness … on a 5-point Likert scale”, which resembles ours.

On the other hand, they are different from ours due to the different objectives. In particular, 
for our binary summarization, we aim to evaluate context relevance and functionality due to the lack of source code.  In contrast, [2, 3] aim to evaluate generated summaries of source code snippets regarding naturalness, adequacy and usefulness. We derive our evaluation prompts from a thorough survey on the reverse engineering domain [4]. The survey summarizes 8 key sub-goals of a human reverse engineer, where two of them (i.e., the overall context of a program, and the functionality of a program) can be enhanced by the binary code summarization task. We therefore construct our evaluation aspects accordingly. 

Appendix C2 in our paper discusses our evaluation aspects for binary code summarization and the rationale. We will revise the title from “Details and Rationale for GPT4Evaluator” to “Evaluation Aspects and Rationale for Human Study and GPT4Evaluator” because our human study and GPT4Evaluator share the same set of aspects.

[4] Bryant et al. Understanding how reverse engineers make sense of programs from assembly language representations. Air Force Institute of Technology, 2012.

> ### but it seems that you lack an informative explanation of why the prober code generated is informative

In this response, we explain why the prober generated code is informative via the two examples shown in the original paper. We will add the corresponding discussions to the paper.

The case study in Figure 11 of the paper shows a function that initializes an encryption key. The decompiled code and the source code are shown in Figure 11a and 11b, respectively. We can see that the decompiled code contains nested loops with complex bitwise operations that are hard to reason about. On the other hand, the prober generates a function “aes_key_expansion”, indicating the original function may be similar to the expansion of an AES encryption key. We can see that the generated code has a similar context to the ground truth source code function and is thus informative to the HOBRE task.
We speculate the prober can generate code within the correct context because it associates subtle patterns (e.g., certain bitwise operations in loops) with encryption.

Similarly, the case study in Figure 12 of the paper shows a function that reads the temperature from a sensor. Two of the code snippets generated by the prober (Figure 12c) are “sht4x_get_humidity” and “sgp30_get_tvoc”. Both code snippets correctly reflect the context that “read data from a sensor”. Note that it is not easy to deduce such context from the decompiled code (Figure 12a). Therefore, the prober provides more context information. 
We speculate the prober captures context information from the code patterns (e.g., consecutive if-statements conditioned on fields of a structure), and the conversion operation (the assignment before the last return statement in Figure 12b). The former may indicate different running status of the sensor, which are commonly seen patterns in cyber-physical systems; and the latter may imply the conversion from integer values to floating point values in certain ranges, which is a commonplace operation when reading data from a sensor.
Thank you for your efforts and new presentation.

It would be more convincing if you could put all the metrics SymLM P, R, F1, charBLEU, METEOR, ROUGE-L, and charROUGE-LSum, and CHRF(charCHRF), G4-F and G4-C for binary summarization or function name generation in one table.
I understand the space limits for the conference paper, but the details of these metrics should be explained in the appendix. 
I recommend emphasizing the two user studies and the conclusions but putting the details of the user study in the appendix.
The new user study in the rebuttal utilizes the context relevance and functionality metric, which is the same as your GPT4 metric (the original user study).
Since the same PhDs conduct the two user studies, would it be fair and no data leakage?
(But I understand the time is limited)

The template of the user study may follow the paper [1][2][3].

The remaining question is mainly for the method and I think the core contribution is the prober.

I understand that new experiments are time-limited, so it would be easier to illustrate the idea with some interesting examples. 
I read the examples you give in the appendix and the rebuttal (some words in bold), but it seems that you lack an informative explanation of why the prober code generated is informative.

Besides, I want to know how you can make the probe more Knowledgeable and Flexible without introducing any new noise.
(maybe some cost analysis in the future)


[1] Improving automated source code summarization via an eye-tracking study of programmers

[2] Automatic Comment Generation via Multi-Pass Deliberation

[3] Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Thank you again for your time and effort. We sincerely appreciate your support!
Thank you for your feedback! We’re pleased to hear that your concerns have been addressed. We sincerely appreciate your support!
We meant that we will clarify in the next version of the paper that these numbers are averaged. 
 
Together with the clarifications we made in the original response, we will perform the following changes.
 
**i. We will change our current discussion of metrics (lines 232-237) to the following.**
 
> We use two sets of metrics to evaluate the performance of a tool for the binary function name recovery task.
> 1. Token-level Metrics. We evaluate the predicted function names following existing work in the reverse engineering domain [1]. The metrics first tokenize both a predicted name and the corresponding ground truth name, and calculate the precision, recall, and F1 score at the token-level. For each metric, we first compute the corresponding scores for individual function name predictions, and then average the scores across all functions.
> 2. Character-level Metrics. We adapt BLEU, METEOR, CHRF, ROUGE-L and ROUGE-LSum for the function name by tokenizing function names into characters and computing these metrics on character level, similar to [2,3]. We call them charBLEU, charMETEOR, charCHRF, charROUGE-L, and charROUGE-LSum. They provide a fine-grained evaluation of the function names and can avoid some limitations of tokenization. Similar to token-level metrics, we first compute the precision, recall, and the F1 score for individual functions, and then average the scores across all functions.
 
**ii. We will add the following sentence to Table 2 caption.**

> Note that we first calculate all metrics for individual functions, and then average the scores across all functions.

**iii. We will integrate the table in our original response (timestamp 07 Aug 2024, Q3) to Table 2 in the paper. The new columns present the results in SymLM precision, SymLM recall, SymLM F1, charBLEU, charMETEOR, charCHRF, charROUGE-L, and charROUGE-LSum.**

**iv. We will change the Appendix C.4 (lines 673-674) to the following**

> Formally, the token-level precision $P$, recall $R$, and F1 are defined as follows:

> $$
P(i) = \frac{\big\| T_{g}^{(i)} \cap T_{p}^{(i)} \big\|}{\big\| T_{p}^{(i)} \big\|}\quad
R(i) = \frac{\big\| T_{g}^{(i)} \cap T_{p}^{(i)} \big\|}{\big\| T_{g}^{(i)} \big\|} \quad
F1(i) = \frac{2 \times P(i) \times R(i)}{P(i) + R(i)},
$$
> where $T_{g}^{(i)}$ is the token set of the ground truth name for the $i$-th test case, and $T_{p}^{(i)}$ the token set of the $i$-th predicted name.

> The precision, recall, and F1 scores for the entire test set are the average scores of individual scores across all test cases. Formally,

> $$
P = \frac{1}{N}\sum_{i=1}^N P(i) \quad
R = \frac{1}{N}\sum_{i=1}^N R(i) \quad
F1 = \frac{1}{N}\sum_{i=1}^N F1(i),
$$
where $N$ is the number of test cases.

We are very grateful for your help in pointing out the places that cause confusion!
We hope the changes have clarified everything. 
 
We will also go through another round of error checking (with fresh eyes) as suggested by the reviewer.

---

## Clarification on how we reuse the scripts

We want to clarify that in Table 2 of the original paper we show the results of charROUGE-L.

CharROUGE-LSum is *another* metric that we additionally added in the rebuttal to show that other character-level metrics are consistent with our original results.

We reused the script of SymLM from their Github repo and the open-source PyPI library “rouge” to compute the SymLM metrics and charROUGE-L metrics for individual function names in our Table 2.


[1] Jin et al. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. 2022 CCS.

[2] Lin et al. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. 2018 LREC.

[3] Shi et al. Natural Language to Code Translation with Execution. 2022 EMNLP.
Thank you for your clarification.

So when I read the paragraph at line 232, how could I get this information? 

Actually, I did check appendix C.4, cited at the end of the paragraph, and I found it talks about the "Precision and Recall Used by SymLM Metrics," which lacks the information (1) SymLM? (2) F1s are averaged?

And if I understand correctly, you further explain that it is charROUGE-LSum but not ROUGE-L in the rebuttal.
How do you reuse the script?
Thank you for your continuous efforts in helping us improve the quality of our submission.

These numbers are correct. Note that we have a triple of precision, recall, and F1 values for each function name (by measuring the individual token or character matches in the name).

The reported statistics are averages over all function names. As such, it is possible that the averaged F1s are not in between the averaged precisions and recalls. A similar case is the Table 2 in this paper [1].

In fact, we directly reused the scripts in SymLM and ROUGE-L to compute such statistics.

We are grateful that you pointed out this confusion. We will clarify.

[1] Allamanis et al. A convolutional attention network for extreme summarization of source code. 2016 ICML
Thank you for your rebuttal. I do respect your effort and clarification.

But one point (I have to point it out since other reviewers are quite confident and positive) is that how you calculate the F1 score with P=50.8, R=53.8 but F1=50.6 in Table 2? (other results similar).

I don't expect to spend such a long time on the evaluation part.

I understand that the "presentation" error may be "fixable," but I have to say these errors stop me from trusting the contribution.

I recommend a rigorous error review for all of us.
Thank you for answering my questions. I am increasing the score to Accept based on the updates proposed in the rebuttal and ethics review.
Dear Reviewer 9NVM,

Thank you again for reviewing our paper and for the valuable feedback. We have made every effort to address your concerns and revised the paper correspondingly. As the rebuttal period is coming to an end, we are eager to know any additional comments or questions you may have. Thank you again for your time!

Sincerely,

Authors
Thanks. I keep my score at 7.
We really appreciate the reviewer’s continuous efforts in helping us improve our submission.

> ### Better suited to SE

With all due respect, we think NeurIPS is the right home for our submission. Even though our task is related to Software Engineering and Security, our core contributions are mainly on the deep learning side, and the NeurIPS community has the sufficient expertise to review our paper. 

Our contributions are:
- We propose a novel and general probe-and-recover framework for HOBRE tasks. The framework has the potential to be useful in other tasks as well. 
- We introduce a novel neural architecture for the prober in the ProRec framework, which is a cross-modal encoder-decoder that encodes binary functions with structural information into node embeddings and conditionally decodes them into symbol-rich source code snippets.
- We introduce the corresponding compute-efficient alignment training for the prober that aligns pre-trained binary encoders and source code foundation models in the token embedding space of the source code foundation models, which is also novel.
- We show that LLM-based automatic metrics have high correlations with human preferences and are suitable for HOBRE tasks, which belongs to a larger topic of evaluation of models.

There are many papers that have been accepted by top-tier AI conferences that focus on SE problems and contribute on the AI side. Here we list a few of them [1-8].

In contrast, many papers published in SE utilize AI techniques/metrics in a black-box fashion, e.g., [9-11]. There is uncertainty whether SE reviewers would appreciate our technical contributions. In the past, we had submitted papers having a similar nature to this submission to SE conferences. Our submissions were rejected because reviewers found our papers difficult to understand and believed that our papers should have been submitted to AI conferences. While we are grateful for the reviewer’s intensive SE expertise, we hope the reviewer could understand that not all reviewers in the SE community would appreciate the technical contributions like those in our submission.  

[1] Gao et al. Virtual Compiler Is All You Need For Assembly Code Search. 2024 ACL Long

[2] Zhang et al. Self-Edit: Fault-Aware Code Editor for Code Generation. 2023 ACL Long

[3] Yu et al. Codecmr: Cross-modal retrieval for function-level binary source code matching. 2022 NeurIPS.

[4] [Oral] Wu et al. Repoformer: Selective Retrieval for Repository-Level Code Completion. 2024 ICML

[5] [Spotlight] Pei et al. Exploiting Code Symmetries for Learning Program Semantics. 2024 ICML

[6] Zhang et al. Coder Reviewer Reranking for Code Generation. 2023 ICML

[7] [Oral] Jimenez et al. Swe-bench: Can language models resolve real-world github issues?. 2024 ICLR

[8] [spotlight] Zhou et al. Docprompting: Generating code by retrieving the docs. 2023 ICLR

[9] Xia et al. Fuzz4all: Universal fuzzing with large language models. 2024 ICSE

[10] Peng et al. Generative Type Inference for Python. 2023 ASE

[11] Zan et al. DiffCoder: Enhancing Large Language Model on API Invocation via Analogical Code Exercises. 2024 FSE


> ### Error review

If we understand the reviews correctly, the errors pointed out by the reviewers could be fixed with simple changes. It does not appear to be the case that our paper is flawed because of these errors. We will perform a rigorous error check as suggested by the reviewer.


> ### The possibility of undetected errors by the reviewers

If we understand the reviews correctly, there were errors that affected understanding. We believe that we have addressed them in the rebuttal. If possible, we would really appreciate it if the reviewer can elaborate on unaddressed errors that caused doubts on our reliability, which seems to be a substantial criticism of our work. 

> ### ”With only part of the user study for these metrics presented in the Appendix. Crucial details are missing”

Our answer in the earlier response under the title “The details of the user study & direct results from user study” provided the missing details. Please let us know if any additional information is needed.

> ### The new user study provided does not offer sufficient evidence to convincingly demonstrate the effectiveness of the method

Our new user study shows that our method is substantially better than the baselines. In particular, ProRec’s improvements over direct-prompting baseline nearly doubles the improvement of RAG over direct-prompting, achieving 4.76/5 in context relevance and 4.62/5 in functionality.

Note that the user study is only for binary summarization. For the other task binary function name recovery, we strictly follow existing work [1] in evaluation and show significant improvement over RAG (5-10% absolute improvement in token-level precision and recall)

[1] Jin et al. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. 2022 CCS.
Thank you for your clarification and the detailed response provided. After careful consideration, I am inclined to maintain my position due to several concerns:

The paper and rebuttal seem to lack a thorough second check, as there are some obvious errors present. This oversight leads me to question whether there might be additional errors that I have not detected. While I acknowledge and respect the positive comments made by other reviewers regarding the presentation, the presence of these errors undermines the overall reliability.

Given my expertise in Software Engineering, I believe that the content might be better suited to a SE conference. The omission of certain details, possibly due to space constraints, suggests that a more targeted venue could allow for a more thorough presentation of your methods and results.

The metrics used in the study are not well explained, with only part of the user study for these metrics presented in the Appendix. Crucial details are missing, and the new user study provided does not offer sufficient evidence to convincingly demonstrate the effectiveness of the method. 

To strengthen your submission, I recommend a rigorous error review, reconsideration of the most appropriate venue for your work, and a more detailed exposition of your metrics and user study results.
We sincerely appreciate your quick feedback on our response.

> ### “Figure 5” in Q2

We meant Table 5 here, thank you for pointing out this typo.

> ### Labeling and explanation of metrics

For binary function name recovery, we leveraged two sets of metrics: character-level metrics and token-level metrics. 

The “(2) precision and recall” at line 234 indicates the token-level precision and recall proposed by existing work SymLM [1]. In Table 2, they are denoted as the “precision” and “recall” under “SymLM”. The “(2)” in the text is a typo. Originally we marked character-level ROUGE-L as “(1)”.  We will clarify this in the revision.

About the “precision” and “recall” listed in Table 2 under “ROUGE-L”, they are the longest common subsequence precision (LCS(X,Y)/|X|) and recall (LCS(X,Y)/|Y|) which are intermediate results of ROUGE-L. As described in the original paper of ROUGE [2], the ROUGE-L score is actually the F-measure of longest common subsequence. To avoid confusion, we will only report the F-measure as the final character-level ROUGE-L in the revision. We will also include other character-level metrics that we discussed in the rebuttal.

[1] Jin et al. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. 2022 CCS.

[2] Lin et al. Rouge: A package for automatic evaluation of summaries. 2004 Text summarization branches out.

> ### Assertions about binary summarization metrics

For “Our user study in Appendix E shows that "only" CHRF is consistent with human preferences.” we meant that **among the commonly used metrics that we measured** CHRF is most consistent. Table 5 in the Appendix shows that GPT4Evaluator also aligns well with human preferences. We will clarify about this in revision.

> ### The details of the user study & direct results from user study

The questions of our user study are similar to the queries to the GPT4Evaluator. Specifically, for each question, we provide a participant with the source code, the corresponding decompiled code, the reference summary, and the summary to evaluate. Similar to the GPT4Evaluator, a participant is instructed to score an evaluated summary from two perspectives (i.e., context relevance and functionality) from scores 1 (worst) to 5 (best).

The user study in our submission involved 12 users and 60 summaries. The users are PhDs / PhD students that either have some background in reverse engineering or are experienced in C/C++/Rust programming. We ensured each summary is scored by at least 3 users, and use the median scores as the results. The questions in our user study were sampled from summaries generated by all three techniques (i.e., ProRec, the RAG baseline, and direct prompting baseline). However, we did not ensure the samples are from the same set of decompiled functions. That is because the goal of the study was to evaluate the metrics, not to compare across baselines. 

To compare different techniques with the user study, following the reviewer’s suggestion, we conduct an additional user study with the 12 users during rebuttal. The additional user study involves 150 questions (50 decompiled functions x 3 summaries from the three techniques). We ensure each question is scored by 3 participants. Moreover, we make sure that summaries for the same function are scored by the same set of participants, so that the relative order across different techniques faithfully reflects human preference.

Below are the average human scores that we collected for context relevance and functionality.

||Context Relevance|Functionality|
|-|-|-|
|direct-prompting|4.29|4.22|
|+retrieval|4.49|4.43|
|+ProRec|4.76|4.62|

We can see that ProRec is the best performing approach with regard to human judgment.

> ### Incorporating the user study into the main body and details in the appendix

We will incorporate the user study into the main body of the paper in our revision.
Thank you for your time.

For me, the rebuttal still presents several points of confusion that require clarification.

Reference Clarification: 

In Q2, you state, "As shown in Figure 5 in our paper,". Should this refer to "Table 5" instead? Please confirm this to ensure that the correct data is being discussed.

Metric Labeling and Explanation: 

In Q3, regarding the metrics discussed on line 234 of the paper, you label these as "(2) Precision and Recall." However, there appears to be no preceding (1), which, if I interpret correctly, might refer to ROUGE-L. In Table 2, metrics for precision, recall, and F1 are indeed listed under ROUGE-L. The connection between these metrics and your labeling is unclear and is not addressed in the rebuttal. Could you provide an explanation to bridge this gap?

Others:

Consistency in User Study References: 

In line 215, I quote, "Our user study in Appendix E shows that "only" CHRF is consistent with human preferences. " Subsequently, in line 218, a GPT-4-based metric for binary summarization is proposed. These assertions seem contradictory and confusing.
The details of the user study is missing in the main paper and the appendix. 

I suggest incorporating the user study directly into the main body of the paper and the details in the appendix since we all agree it is a crucial component of the study regarding meaningful metrics for the binary summarization task.

Why not directly conduct user study for your techniques?

I am willing to consider increasing my score if we could address these confusions.