Title: Transforming Generic Coder LLMs to Effective Binary Code Embedding Models for Similarity Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: cybersecurity

embedding

binary code
Abstract: Cybersecurity and software research have crossed paths with modern deep learning research for a few years. The power of large language models (LLMs) in particular has intrigued us to apply them to understanding binary code. In this paper, we investigate some of the many ways LLMs can be applied to binary code similarity detection, as it is a significantly more difficult task compared to source code similarity detection due to the sparsity of information and less meaningful syntax. It also has great practical implications, such as vulnerability and malware detection. We find that pretrained LLMs are mostly capable of detecting similar binary code, even with a zero-shot setting. Our main contributions and findings are to provide several supervised fine-tuning methods that, when combined, significantly surpass zero-shot LLMs and state-of-the-art binary code similarity detection methods. Specifically, we up-train the model through data augmentation, translation-style causal learning, LLM2Vec, and cumulative GTE loss. With a complete ablation study, we show that our training method can transform a generic language model into a powerful binary similarity expert, and is also robust and general enough for cross-optimization, cross-architecture, and cross-obfuscation detection.


Peer reviews:
comments: SUMMARY:
Vulnerability detection in binary code is a very challenging and the generic large language models are not yet mature enough to reason about similar code. This work proposes uplearning the pretrained LLM to identify the binary code similarity and has shown effectiveness against zero-shot LLMS and the state-of-the-art binary code similarity detection methods.

STRENGTHS AND WEAKNESSES:
Strengths :
The causal relationship is vital in reasoning the binary code, specifically in understanding the flow. This work is uptrained the LLMS with the causal training which can improve the LLMs capabilities against binary code. 

Weaknesses: 
I do not see any specific weakness in this work

QUESTIONS:
The equations 1 & 2 are not explained

LIMITATIONS:
Limitations are included under the conclusion. Limitations include the quantity of the training data set compared to previous work, and the lack of potential downstream tasks, etc. However, this work can be good initial work achieve these limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores using LLMs for binary code similarity detection. It demonstrate that pretrained LLMs can detect similar binary code even in a zero-shot setting, and proposes supervised fine-tuning methods, including data augmentation, translation-style learning, LLM2Vec, and cumulative GTE loss, that significantly outperform zero-shot LLMs and state-of-the-art approaches. The approach enhances robustness across different optimizations, architectures, and obfuscations, transforming generic LLMs into effective binary similarity experts.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n+ The evaluation includes diverse scenarios from multiple perspectives (e.g., cross-optimization, cross-architecture, cross-compiler, cross-obfuscation). \n\n## Weaknesses\n- The generality of the approach is not well justified empirically given only one small backbone model is involved. \n- The downstream application tasks of the BCSD techniques are missing (in terms of both the discussion and empirical evaluation). It hurts the significance of interpreting the metric improvements of the proposed approach in practice.\n\nQUESTIONS:\n1. How does the backbone model (Qwen2.5-Coder-0.5B) perform without training? (This question is concerned to show the improvement from the approach)\n\n2. How does the approach perform when applied to different backbone models with different scales of parameters? (This question is concerned on the generality of the approach as currently the evaluation is restricted on one small backbone model).\n\n3. What are the downstream application scenarios of the BCSD techniques in practice? (This question is related to the importance and motivation of the studied program, also, it could help justify why the BCSD can be formulated as a retrieval problem). \n\n4. How is the retrieval pool constructed? Why use the size 1000 by default?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of training embedding models for detecting if a pair of code binaries are similar, i.e., generated by compiling the same source file but with different compiler options or architecture backends. For this purpose, the paper presents a method for fine-tune existing code models using a combination of causal language modeling, masked language modeling, and contrastive learning objectives. A model (Qwen2.5-Coder-0.5B) fine-tuned with these objectives is compared with various baselines on similar binary code retrieval task. Experiments show that the fine-tuned model outperforms various baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The empirical results are very promising.\n\n2. The proposed training methods are fairly straightforward, which is a good thing! It seems like the methods should be easily applicable to other baseline models as well.\n\n### Weaknesses\n1. It would have been nice if the experiments included results from fine-tuning another model. At the moment, one cannot say for sure if the presented method will show improvements on other models as well.\n\n2. The experiments only consider similarity between pairs of binaries that are compiled from the same source code. I am very curious what the results look like for a pair of binaries that are compiled from different, but semantically equivalent source codes.\n\n3. If I understand correctly, during inference, the method requires access to metadata about the binaries such as the hardware architecture, the compiler optimization levels, etc. Can such information be extracted from binary files? If not, the method may not practically applicable.\n\nQUESTIONS:\n1. Why not consider pairs of binaries that are compiled from different, but semantically equivalent source codes?\n\n2. Does the method require metadata about the binaries such as the hardware architecture, the compiler optimization levels, etc? If so, can this information be extracted directly from binary files? If not, how will the method be used in practice?\n\n3. Equation 6 is not clear. What do you mean by \"multiple distributed models\" and \"accumulated inputs\"?\n\n4. Does the method require disassembling the binary first? Section 3.1 seems to suggest this. It would be good to clarify this.\n\n5. Which dataset is used for results in Fig 3?\n\nLIMITATIONS:\nThe paper has a brief mention of limitations in the conclusion. Perhaps, this could be expanded. One limitation that could be discussed is whether the method is applicable for two binaries that are compiled from different but semantically equivalent source files.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training strategy to transform an LLM into an assembly code embedding model. It flattens the code and adds structural \"language\" tokens for improved syntax and semantic representation. It treats function pairs like translation tasks to learn cross-variant semantics. It leverages LLM2Vec to enable bidirectional attention in causal models for better embeddings. It also utilizes cumulative GTE loss that enhances training with limited resources. Experiments are conducted on two datasets, and results show that it outperforms baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Overall, it's well-written and easy to follow.\n2. The evaluation considers multiple datasets and baselines, and also includes ablation studies on different components.\n3. The results show that the proposed method outperforms baselines.\n\nWeaknesses:\n1. It would be better to include the time cost. Because this paper tries to explore an efficient fine-tuning method, it's suggested to show that the proposed method is more efficient than existing methods.\n2. It would be better to evaluate some other cases. \n\n    2.1 If the pool doesn't contain \"exactly one similar function\", that is, \"no similar function\" or \"multiple similar functions\", what will be the precision and recall?\n\n    2.2 If the user doesn't know the language information of the binary code, or wrong language information is provided, what will happen?\n\n    2.3 When the training configuration is different from the test one, how will the performance change? For example, if the model is only trained on o0, o3, can it work for o1, o3?\n\n3. It would be better to introduce and explain the symbols in each equation. For example, $q_i, k_i, k_j, \\tau, \\eta, (n)$, etc.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nIt may not work when the binary code cannot be disassembled correctly.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the task of binary executable code similarity detection with help of LLMs. They propose finetuning an LLM model on assembly data and show that this finetuned model outperforms similar models evaluated in zero-shot setting.\n\nSTRENGTHS AND WEAKNESSES:\n**Weaknesses**\n\n- The paper is poorly written; the motivations behind decisions made in the study and proposed novelty are not spelled out; some of the contributions mentioned, such as \"comparing baselines\", are not sufficient to be considered a contribution in my opinion. There are multiple ambiguous pronoun references which affects the clarity of the paper, e.g. in Conclusion; acronyms are used without prior introduction, e.g. what is AR on line 139? GTE on line 173?\n\n- Key details are missing, e.g. on line 126 the authors say: \"we add several additional tokens to indicate the “language\" information of each binary function, which corresponds to the optimization level, compiler, obfuscation, and architecture. Adding such information has proven to be effective, especially for cross-architecture detection.\" - without specifying what tokens, how and where they are added, at what stage, etc, this information is not as useful for the reader as it could be otherwise. In general, the paper does not seem to be written with AI professional readers in mind, e.g. the authors go into details explaining how autoregressive models learn, but gloss over what their used obfuscation methods are. \n\n- The authors suggest that their proposed solution based on continual training and finetuning of an LLM model is cheap and effective, which is questionable considering approaches like parameter-efficient FT or inference-time demonstrations with larger models are not considered at all. \n\n- The choice of the baseline models and how each one is trained is unclear to me. The authors mix embedding models, such as CodeT5+ and GraphCodeBERT, with generative models, such as Qwen2.5-Coder1.5 for a task of ranking pairs. How are they compared is not clear. This weakens the results because it is not clear what comparisons are being drawn and which of the proposed modifications/design choices are responsible for performance report.\n\n- The prior work section is incomplete when it comes to recent developments, and omits most recent works and models, instead referring back and comparing to 5+ years old models, such as GraphCodeBERT and CodeT5+, which are not state of the art, and weren't intended to be used with assembly either, so not clear why those were chosen over other more modern & stronger code models. To make the paper and it's findings valuable it would need to draw comparisons to more recent models such as DeepSeek models, SantaCoder, WizardCoder, CodeGemma, CodeLlama, etc.. The authors use Qwen-Coder as their base model, which is a strong, state-of-the-art model.\n\n- The motivation behind performing continual training on a generative model and using it as an embedding model is not clear, and most of the design choices are not intuitive, and neither are they explained or supported, which limits the usefullness of the paper since the takeaway messages are not clear. It is also not at all surprising that finetuning an LLM may outperform other, similarly-sized LLMs that are evaluated in zero-shot manner, so the whole evaluation setup needs an overhaul to provide new knowledge for the readers. \n\n\n**Strengths**\n\nThe authors use a comprehensive dataset of cross-architecture, cross-optimization, cross-compiler, cross-obfuscation binaries for their experimentation.\n\nQUESTIONS:\n- \"the same source code can be compiled into many formats of binary code using different compiler settings and environments, including optimization levels, machine architectures, compiler software, and obfuscation techniques. \" - The same is true for source code similarity detection, the same functionality can be written in many ways in source code. While this is a limitation and potential challenge, further motivation is needed to understand why the LLMs that work for source\n\n- On line 193: \"As our model is based upon Qwen2.5-Coder-0.5B, the Qwen2.5-Coder-1.5B baseline is a good candidate for ablation study.\" - not clear why you suggest it is an especially good candidate compared to any other coding model? Why that size of the model?\n\n- Authors mention they finetune all models with their contrastive loss, except Qwen-Emb, which is supposed to illustrate the difference between code & non-code models, but that makes no sense to me. Why also not finetune Qwen-Emb and show that code models perform better compared to non-code ones, even when finetuned? Or perhaps authors meant something else, but it wasn't clear to me.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-04"}