Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning
Authors: Nan Jiang
Chengxiao Wang
Kevin Liu
Xiangzhe Xu
Lin Tan
Xiangyu Zhang
Petr Babkin
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
hierarchical attention
contrastive learning
assembly code
Primary area: foundation or frontier models, including LLMs

Abstract: Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 -- 21.58% higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks.

Published as a conference paper at ICLR 2025
NOVA: GENERATIVE LANGUAGE MODELS FOR ASSEM- BLY CODE WITH HIERARCHICAL ATTENTION AND CON-
TRASTIVE LEARNING
Nan Jiang
Chengxiao Wang
Kevin Liu
Purdue University jiang719 @ete711@purduee.o
University of Illinois Urbana- -haaaa-haaapagg cw 124 il4@ilinois.edu
Lynbrook High School kevin.bx.liu gmail.com
Xiangzhe Xu
Lin Tan
Xiangyu Zhang
Petr Babkin
Purdue University
Purdue University lintan @ purdue.edu
Purdue University xyzhang ccs es. purdue..euu
J.P. Morgan Al Research
xu1415@ @ul4es@pardue.euu
petr.babkin @omooign@ppmoaooomm
ABSTRACT
Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations assembly code. To overcome these challenges, this work proposes hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively, and designs contrastive learning objectives train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques binary code decompilation by up to 14.84 21.58% (absolute percentage point improvement) higher Pass @ and Pass @ 10, and outperforms the latest binary code similarity detection techniques by up to 6.%7% Recall @ 1, showing promising abilities on both assembly generation and understanding tasks.
INTRODUCTION
Binary code plays an irreplaceable role in the security domain. being the foundation of crucial tasks
including vulnerability detection (Güler et al.) [2019] Duan et al. 2020 Chen et al. 2022b), malware
detection (Spensky al., 2016 Aonzo al. 2023 Xu et al. 2014), binary recovery (Su et al.
2024
Zhang et al. 2021 Chen et al. (022c), and legacy software maintenance (Carbone et al. 2009
Carlini al. 2015 Martin et al. 2010). For example, when performing tasks such as identifying
attacks and malware, security analysts often only have access to assembly, i.e., the human-readate representation binary code, which extremely difficult to understand (Su et al. 2024 Zhang ct al. 20221] Chen et al. (202cc). Thus, combined with the increasing sophistication of cybercrime that poses significant threats worldwide (c.g., cybercrime is predicted to cost the world S10.5 trillion annually by 2025 (Sausalito 2020 effective binary analysis techniques are in high demand.
einncTude <stdio.h> siinclude <math.
endbr64
endbr64 mov
4:
push
%rbp Srsp. Srbp
(8rdi) Secx (Krsi) ) Sedx
S:
mov
6:
mov
int compare(int *x, int *y) {
8:
Srdi. -xx -8xb(
88
SextIitiiIz Seax
if (*(int*)x < *(int" (y)
Arsi ; 00x10 ((bbb))
kedx, Seex
1f (*(int*)) x *(int
return -1;
116
14: nov
mov
-8x8(Srbp), (krax), Sedx
0xx00 Oxbp(Krbp)) Srax
11:
14:
mov2bl
jL
setg
Kal
Kal, Reax
return
:la mov
(krax), Seax
17:
retq
return B:
Me:
Seax Sedx
Source Code Function
Assembly (00-Optimizel)
Assembly (O1-Optimized)
Figure 1: Example that shows the semantics and diverse optimizations of assembly code.

---

Published as a conference paper at ICLR 2025
Large language models pre-trained source code have brought improvement in various software development domains (Chen et al.
2022a Liu et al. 2023a Chen et al. 2023; Le et al.) 2022
Jiang al.] 2023 Xia al. 2023) However, these LLMs are not designed for or traied with assembly corpus, not achieving their full potential on binary code analysis tasks such as hinary code similarity (Wang et al. 2022 Xu al. 2023a), malware detection (Su al.] 2024), and binary code decompilation (Tan et al.) 2024 Armengol-Estapé et al., 20244 Hosseini & Dalan-Gavitt (022).
Existing work applying LLMs on assembly code mainly piggybacks on encoder-style LLMs (Wang et al. 2022] Su et al. 2024 Xu et al. 2023a), unable to benefit from the more extensive pre-training, updated architectures, scaling of state-of-the-ee generative LLMs. Other work using generative LLMs for decompilation shows a low unit test passing rate of the decompiled programs (Tan et al.
2024
Armengol--ssaape al., 2024).
The challenges of leveraging generative LLMs for assembly code are twofold. First, compared to source code, assembly code has a lower information density. short source-code sequence maps to assembly-code sequence that is often several times longer. Thus, assembly semantics span across a long sequence of tokens. Figure [ (a) shows an example of a source code function that compares two integers, while Figure shows its corresponding assembly code optimized with 00 flag. In the 0p-opmimieed assembly code, the five instructions from 10: mov -0x8 (Xrbp), &rax to 1e: cmp Xeax, %edx perform the checking whether the value of is smaller than the value of y (correspond to if (*(in**) < * (int*)y) in the source code). A single assembly instruction alone represents little meaningful semanties in the source code. It is the combinations of many instructions and the dependencies between them represent the semantics. Such combinations of instructions are long, which hard for LLMs learn.
Second, assembly code is diverse due to compiler optimization. The assembly code of the same source code function looks dramatically different with different compiler optimization. Figure [ (c) shows the assembly of the same function compiled with 01 and 0e flags, which consists of significantly different set of instructions. Such syntax diversity is hard for LLMs to learn, preventing LLMs from obtaining consistently good performances on differently optimized assembly code.
In this work, we develop Nova, generative foundation LLM pre-trainee rrettrained for assembly code with two key novelties. First, to address the infformationddense and long-sequence challenge, we design a hierarchical self-attention, which contains three categories of attention at different levels of granularity: intra-instruction attention, pecceding-instructinn attention, and inter-instruction attention. The key insight is to build attention summaries, i.c., we create per-statement attention labels, which act the summary of a statement. We then use innttritnntttttttioon attention to capture semantics between a token and its preceding instruction label and use inter-instruction attention for long dependencies. Besides, we design functionality contrastive learning and optimization contrastive learning objectives to train Nova to learn the semantics behind the diverse syntax of assembly.
This work makes the following contributions:
We propose a novel hierarchical attention mechanism that captures the assembly's low-density semantics three granularity levels.
We design contrastive learning objectives to train LLMs to learn assembly with diverse optimiza-
tions and encode assembly more efficiently.
We develop Nova, a generative foundation LLM with hierarchical attention and contrastive learning for assembly. Nova outperforms state-of-the-a -ortte--rttttttee on binary decompilation by up to 14.84 21.58% higher Pass @1 and Pass @1 10, and on binary similarity detection by up to 6. 17% Recall @ 1.
Availability: we release Nova models at al/nttss:/hhuginggaaa wttp:///ggiina.acccoooooetaaeeee co/lt-asset/nova 1. 3b and https
7 i/uuggiggface wwwpggaiinnaaee.oottaststte/teee wttp///aaaaseee/oaaaa...777
2
RELATED WORK
2.
BINARY MODELS
Machine learning models are widely used in binary program analysis tasks. However, these models
typically designed for specific tasks such binary code similarity detection (Pei et al. 2020 Xu
et al. 2023a Wang et al. 2022 Xu et al., 2017 Ding et al. (2019). variable name prediction (Chen
et al. 2022c Xu et al. 2023b Zhang al. 2021 He al. 2018 Lacomis al. 2020). binary
2

---

Published as a conference paper at ICLR 2025
code type inference (Pei et al. 2021), and Chen et al. (202d)) Liu et al. (003bb; Hosseini & Dolan-Gave 2022).
Recent techniques have started to pre-train foundation LLMs for binaries. CodeArt (Su et al.
2024)
pre-trains encoder-style LLMs with a regularized attention design to better encode assembly code semantics. SLaDe (Armengol-Estapé et al.] 2024) trains BART (Lewis al., 2019) models on assembly. Meta LLMCompiler (Cummins et al. (2024) train CodeLlama models on LLVM IR to optimize binary code. LLM4Decompile (Tan et al.) (024)) trains DeepSeekCoder with assembly for binary code decompilation. However, AodeArt does not generalize to generation tasks due to its encoder architecture. LLMCompiler trained on LLVM IR cannot be effectively transferred to assembly code. SLaDe and LLM4Decompile are limited in performance due to lack of special designs for assembly. In contrast, Nova proposes hierarchical attention and contrastive learning objectives, outperforming existing techniques on both understanding (binary code similarity detection) and generation (binary code decompilation) tasks.
2.2
LARGE SOURCE- COURCE-CODE MODELS
LLMs demonstrate promising results on many code-related tasks, such as code generation (Chen
et al.
2022a; Liu et al.) [2033a] Chen et al. [2023] Le et al.] 2022; Yue et al.] [202]] Chen et al.
202T
Nijkamp et al.] 2022; Fried et al. 2023] Rozière et al. [023] Guo et al. 2024 Lozhkov
et al.
2024] Hui et al.
2024). bug fixing (Jiang et al. 2023 Xia et al. 2023) and vulnerability
fixing (Wu al., 2023] Steenhoek et al. 2023) He & Vechev (023). The advances in using LLMs
are attributed to the knowledge learned from massive source code and natural language text in their
training datasets (Touvron et al. 2023; OpenAl 2023). Nova is designed and trained for assembly,
which has unique challenges such as low information density and diverse optimization.
2.3 ATTENTION MECHANISM
Standard self-attention is widely used in transformer architecture Vaswani et al. 2017) to capture soft dependencies between tokens in the input. Many special attention mechanisms have been designed for better learning in various scenarios (Yang et al. 2016 Huang et al. (2024). LongCoder (Guo et al., 2023) combines window attention and global attention (attention sink (Xiao et al. 2024 to handle long input of source code. We have shown that LongCoder's 's window attention less effective than Nova's on assembly code. CAST (Shi et al. 2021) is a new neural architecture that splits the abstract syntax tree (AST) of source code into subtrees, encodes the subtrees, and aggregates to the final encoding. PA-former (Chai & Li] 2024) is a new neural architecture that constructs source code as pyramid input based their AST structure and contains pyramid attention mechanism to calculate the features in hicrarchical way. HierarchyNet (Nguyen et al. 2023) is neural architecture that considers source code AST, data flow, and control flow graphs. Similarly, it cannot be applied to assembly code. Different from CAST, PA-former, and HierarchiyNet, Nova's attention design is for assembly code, is more lightweight and can be plugged into any pre-trained generative LLM.
APPROACH
Figure presents the overall approach of Nova. We build Nova on top of foundation models for source code (Guo al. 2024) utilize their source code and natural language generation ability. We first collect large assembly corpora (Section 3.1). Section 3.2 describes Nova's 's hierarchical attention design. With the collected assembly corpora, we then pretrain Nova with language modeling and contrastive learning objectives (Section 3.3). Then, we fine-tune Nova on two important downstream
tasks, binary code decompilation, and binary code similarity detection (Sections 3.4 and 3.5 to
prove Nova's 's effectiveness and benefits to the binary research domain.
Pre Prei-Tained LLM
Functional Contrastive
Table Statistics (number of C and X86-64 as- sembly functions) of the pre-training datasets.
Learning
Language Modeling
Nova
Optimication Contrastive Learning
Data Hicrurchical Attention
Origin
Functions
00
02
AnghaBench
757.1K
743.1K
726-4K 718.7K 717.8K 3.7M
Figure 2: Overview of developing Nova
Stack
138.8K
125.1K 119.7K 116.9K 108.8K 609.3K
3

---

Published as a conference paper at ICLR 2025
3.1 DATA COLLECTION
In this paper, we focus X86-64 assembly functions for C programs. Yet, Nova's approach is generalizable to other assembly languages such as ARM assembly.
We derive our X86-64 assembly functions dataset from two source code corpora: C functions in The-Stack (Li et al. 2023) and AnghaBench (da Silva et al. 2021). We compile the C programs into executables using gec with different optimization levels (i.e., 00, 01, 02 and 03), strip the executables to remove debug information, and disassemble them into X86-64 assembly code using ob jdjupp. We treat every function as separate data point. Table T shows the number of C functions in the two original datasets, and the number of X86-64 assembly functions we collected from them.
We perform certain normalization on the assembly functions: (1) removing all the "x" and comments,
(2) adding whitespace around "," ,"" "(", ">", : (3) converting all the hexadecimal numbers to decimal numbers, and (4) replacing the address of each instruction with special labels (e.g., replacing "ø" and "4" in Figure (b) with "[INST-1]" and "[INST-2]") placing at the end of each instruction. More details are Appendix A.
3.2 HIERARCHICAL SELF- ATTENTION
Nova uses hierarchical self-attention that is specially designed to learn the Dov-inorrmation-density semantics the long sequence of assembly code. Specifically, Nova learns the assembly code in an hierarchical way by providing modified attention mask. Different from standard token-level
attentions (Vaswani et al. 2017: Radford & Narasimhan 2018: Radford et al. 2019 Brown et al. 2020). our hierarchical attentttniion contains three categories at different levels of granularity.
asserbly code:
edd
attertion
[KST]]]
1AST
KC
1NST
[KST4]
Assembly Code Hierarchical Artention Mask
(N530
OX
AX
LIST
TDV
VMX
PWV
[hST-7]
TDV ebx
TDV WIX
the
LNST
wixx
ebo
Tokenized Assembly Code
Attention Illustration
Assembly Code Atteation Compatible Standard Attention
Figure 3: Design of Nova's hierarchical attention for assembly code
(1)
Attention: Due to the low information density in assembly, intra-instruction attention is designed to capture the summary of every instruction, which is the standard causal attention but limited to tokens of each instruction (the yellow part in Figures 3 (a) and (b). Tokens in different instructions have attention weights. The "
" label the end of the instruction has attention to all the tokens in the instruction and thus captures the semantics the entire instruction (e.g., "[INST-1]" captures the semantics of "mov eax, $1").
(2) Preceding- Instruction Attention: In addition to the local semantics of each instruction, the use of assembly instructions (such the choice of registers) depends the context. For example, after
4

---

Published as a conference paper at ICLR 2025
the first instruction "mov eax, $1", the second instruction should not reuse "eax" to store another value "s2" immediately. To capture such context, the preceding-ines instruction attention enables each token in an instruction to have attention to the "
" label of the preceding instruction (the light green part in Figures 3 (a) and (b)).
(3) Inttruentttnttion Attention: To understand function semantics (i.e., functionality), which lies in the dependencies across different instructions, the inter-instruction attention is designed to let the "
" label of each instruction have attention to all the labels of previous instructions. For example, "[INST-4]" has attention to "[INST-1J", "[INST-2]", and "[INST-3]" (the dark green part in Figures 3 (a) and (b)). The inter-instruction attention is only enabled for "
" labels, as they represent the semantics of each instruction.
To sum up. the hierarchical self-attention splits assembly code semantics into three levels: intra- instruction attention captures instruction summaries, from the preceding instruction, and inter-instruction attention models long dependencies across instructions using "
" tokens that contain the instruction summary. Figure highlights the compatibility of Nova's hierarchical attention with standard self-attention for text and source code. Leveraging the proven performance of standard self-attention in existing LLMs, we retain the causal attention mask within and across chunks of text or source code (shown in light grey in Figure 3 (c)). Cross- attention between text, source code, and assembly is restricted to "
" tokens, which encapsulate assembly instruction summaries.
instruitionn attention provides context
3.3 CONTRASTIVE LEARNING
The syntax gap between assembly code and source code, and syntax diversity between differently- optimized assembly code make LLMss struggle to distinguish the semantics behind the syntax. Nova adopts contrastive learning technique (Gao et al., 202T) during pre-training to train LLMs to encode assembly code meaningfully w.r.t semanties. The standard pre-training objective is language modeling by minimizing the negative likelihood of code in the pre-training corpus (Radford & Narasimhan 2018), notated Ltm- In addition, Nova is pre-trained with two new objectives, L for functionality contrastive learning and Lost for optimization contrastive learning.
func comps
comple Beseentle 00, 002 00, O0,
Lung <emp)
Aucy <oorb S00g
Uhe
vert
Nuc, amins -5Cg
scer me!
de ong. ong?
di dfferent functions
Junc cmaxs
Ctein
distance matrix drll
(a) Ccross Functionaliiies
(b) Design F'ucctionallty CL
(0) Ccrosss Optimizations (d) Design Optimization CL
Figure 4: Design of functionality and optimization contrastive learning (CL). "asm" denotes assembly.
Functionality CL: Functionality CL trains Nova to focus more on the functionalities of assembly code rather than the syntax. Code with the same functionalitty (assemblies from the same source code), should be encoded closer the latent space. For instance, Figure 4 (a), embeddings of source and assembly code of function "'mm"" are closer to each other, and the same for function "sort".
Nova is designed and implemented on decoder-only generative LLMs, and we refer the hidden states from the last transformer layer embedding. For source code, we use the average of each token's 's embedding as the source code function's is embedding. For assembly, we use the average of all the "CINST]" tokens' embedding the embedding of the assembly function, each "
" token is supposed to capture the semantics of that instruction by the design of our hicrarchical self-attention.
Let ey be the embedding of function ff in form (s -1 for source code, and E [0, 1, 2, 3] for 00 to 03 optimized assembly). For simplicity, let S = [-1, 0, 1, 2, be the domain of s. Functionality CL optimizes Nova's 's embeddings to satisfy the constraint:
Vf. F, maz (dlej, S.E
<
min s.ES (d(ej,,ei,)
f
5

---

Published as a conference paper at ICLR 2025
where d calculates the ly distance between two embeddings and F is the full set of functions in the training corpus.
The embeddings of batch of functions, each represented in two different forms, can be optimized to satisfy these constraints. For the example in Figure there are two forms (source code and oo assembly) of four functions. Once Nova encodes the batch of source code and assembly functions, we calculate the distance matrix (Di))tuffer = {d(e), ei ,, and minimize the loss:
exp (d(e),,
Ljd = -log Σ EE A,ESS SEE
Σleer Exp exp (die).
This objective minimizes the distance between the embeddings of the same function, which is the diagonal in the distance matrix. practice, this loss is computed and back-propagated for each batch of functions. Given a batch size of 64, F represents set of 64 unique functions in the batch.
Optimization CL: LLMs can be confused if being asked to directly connect a source code function its 03-optimized assembly, due their dramatically different syntax. Such a huge gap can be filled by learning how the source code is transformed to 00, 01, 02 and eventually to 03 assembly, as the optimization levels are ordered. Higher-level optimization applies super-set of optimization rules compared to lower-level optimization.
Nova learns such order with the optimization CL objective, encoding differently-optimiin assembly code orderly. Optimization CL optimizes Nova with the constraint: the more optimizations applied, the larger the difference between embeddings of optimized and unoptimized code. For instance, Figure (c) and (d) illustrate that for the same function "cmp", the distance between source code and assembly increases when the optimization level increases. Formally, optimization CL minimizes the following loss:
Troel Σ Σ max (0, d(e), d(e),e
TEF Acty
Overall, the final training loss combines the three: L = Lim + X(Lrget + Loaa), where A to 0.1 to balance the losses in this work.
3.4 TASK 1: BINARY CODE DECOMPILATION
Binary code decompilation (BCD) helps developers to understand binary code by recovering binary code into more readable high-level source code (c.g.. C programs) (Fu et al. 2019 Liang et al. 2021 Anmengol-Estapp et al., 2024 Tan et al.) 2024). The input the model for BCD formatted as an instruction prompt (notated by p): # This is the assembly code with {opt} optimization: (asm). where "opt" the optimization- decompile. Nova is fine-tuned to generate the expected source code function src following the instruction prompt. The fine-tuning objective minimizing the loss: Lied log P(src p).
applied to the assembly and "asm" is the assembly code to
3.5 TASK 2: BINARY CODE SIMILARITY DETECTION
Binary code similarity detection (BCSD) aims to measure the similarity between two binary code snippets (Wang et al.) 2022] Su 2024), which is the foundation various applications such as plagiarism detection (Luo et al. 2014 Sæbjornsen et al. 2009) and vulnerability detection (David & Yahay 2014 David et al. 2018 2017 2016).
A widely used setting is taking query assembly of the function fo that is compiled with one optimization level (denoted by s), and a pool of candidate assembly of K (e.g., 50, 100, etc.) functions (notated by J?, < < K) compiled with different optimization level (denoted by t + s). There exists a unique candidate assembly coming from the same source code as the query (311 Sig K, f. = f?, called the positive candidate). Nova is fine-tunce to encode these binaries, so that the positive candidate has the highest similarity with the query assembly among the pool. The learning objective is follows:

---

Published as a conference paper at ICLR 2025
exp (diejrn c(p)
LECSD log
Σ
LiSisk cop (dlegnse/z))
ISiSK
We follow previous work (Su et al. 2024) to be 00-assembly and 03-assembly, which the hardest setting.
EXPERIMENTAL SETUP
This section describes the setup of pre-training and fine-tuning of Nova, as well as the existing baselines we compare Nova with, and the evaluation metrics we used in the two downstream tasks. Appendix A.2 contains additional details such as training hyper-parameters, and evaluation setup.
4.1 PRE-TNAINING
We the C and X86-64 assembly functions collected from AnghaBench and The-Stack for pre- training. We pre-train Nova starting from DeepSeek-Coder (Guo et al. 2024). and the hierarchical attention is applied on half of the attention heads to balance between its effectiveness and the existing knowledge in the standard attention layers (Justified in Appendix A.3). Nova is pre- pre-trained with language modeling for one epoch, followed by contrastive learning objectives for another epoch.
4.2 FINE-TUNING FOR BINARY CODE DECOMPILATION
Training Data: We sample (due to computation resource limitation) 2.16M -onsubblyt--ssuurc-ooode pairs (0.338B tokens) from the pre-training corpus build the BCD fine-tuning data.
Test Data: We use SeataanEval-Decomppe (Tan et al. 2024) as the test benchmark, which was used in training. HlumanEyal-Decomen
is derived from the language adaptation of the HumanEval (Chen et al. 202T) benchmark and contains 164 C functions, each compiled with 0e - 03 optimization flags and disassembled into X86-64 assembly.
Baselines: Nova is compared with existing SOTA LLM4Decompile (Tan et al. 2024), Meta LLM- Compiler (Cummins et al.) 2024). other open-sourced general code LLMs (Rozière et al. 2023 Lozhkov et al.) 2024| Guo et 2024, Hui et al. 2024 Mishra et al., 2024 Guo et al. 2003)) and commercial LLMs GPT-3.5- Turbo and GPT-4o). LLM4Decompile trains DeepSeekCoder using the same AnghaBench corpus for binary decompilation. Meta LLMCompiler trains CodeLlama models using LLVM IRs, X86, and ARM assembly code to optimize and translate binary code.
assembly function, using the temper- ature of 0.2 and top_p of 0.95 (Chen al. (02T). Except for I.L.4Decompile and Nova that are provide all other basclines with three-shot examples for few-shot learning (Brown et al. 2020). The decompilations are executed with the test cases to verify reported.
Evaluation: We let each model sample 20 decompilations
fine-tuned for binary code decompilation,
the functional correctness. Finally, Pass @ and Pass @ 10 (Chen et al. 2021
4.3 FINE-TUNING FOR BINARY CODE SIMILARITY DETECTION
Training Data: compare Nova with existing works on BCSD fairly (Wang et al., 2022; Su et al., 2024), we use the same dataset, BinaryCorp-3M (Wang et al. (022). as the fine-tuning data for BCSD, which contains the 00 and 03 assembly 224,606 functions.
Test Data: Following existing work (Su et al. 2024 Xu et al. 2023a), use rorldworld benchmarks, Binutils, Curl, ImageMagick, SQLite, OpenSSL, and Putty, as the test benchmarks, which are nonexistent in the training data.
Baselines: Nova is compared with jTrans (Wang et al. 2022). DiEmph (u et al. 2023a) and CodeArt (Su et al. 2024). jTrans is Transformer (Vaswani al., 2017) encoder trained on binaries with masked token prediction and jump target prediction tasks. DiEmph uses an instruction deemphasis technique prevent the model from learning instruction distribution biases introduced by
7

---

Published as a conference paper at ICLR 2025
compilers. CodeArt proposes a regularized attention mask for encoder models to capture instructional semantics and data dependencies.
Evaluation: We randomly sample K 50, 100, 200, 500 source code functions from each project, compile them into binaries with 08 and 03 optimization flags, and disassemble them into X86-64 assemblies. BCSD techniques encode these assemblies into embeddings (Nova uses the average last-layer hidden states of all the "
" tokens in an assembly as its embedding). Then each oe assembly is used the query calculate their similarity with the K 03 candidate assemblies. Metric Recall @ is reported as the ratio of queries for which the candidate from the same source code has the highest similarity among all the candidates.
RESULTS
5.1 BINARY CODE DECOMPILATION
Comparison with SOTA Techniques: Table shows the Pass@1 and Pass@ 10 of the decompiled code from assemblies on HumanEEvl--ecompllc The results are grouped by optimization level (i.e., the benchmark contains 164 assemblies of each optimization level), and the average is also reported.
Table 2: Nova's Pass @ K and comparison with existing techniques on Human -eeioppppeee..........
Pass I
Pass 1O
Techniques
00
01
02
03
Avg.
00
O2
Avg.
Codel Jama-7
6.95
3.81
4.54
3.78
4.77
8.53
5.97
7.34
5.17
6.75
StarCoder2-7B
5.64
5.95
5.56
5.18
6.09
7.17
6.80
DeepSeekCoder
9.63
7.56
7.41
6.68
7.82
13.60
11.38
11.52
9.44
11.49
Owen-2.5-Coder-7B
4.76
5.27
7.69
6.56
5.79
LLMCompile-778
5.95
5.85
5.55
5.82
5.79
7.01
7.31
7.47
7.01
7.20
GPT-3.5-Turte
7.41
6.13
4,33
3,90
5.44
9.56
8.38
6.23
5.12
7.32
GPT-4o
21.34
18.29
14.48
13.05
16.79
29.94
26.74
21.42
19.88
24.50
LLM4Decompill-1.3
15.30
Nova-L33
37.53
21.71
22.68
18.75
25.17
49.38
34,84
36.95
32.03
38.30
LLMrDecmmpie-6.77
29.97
19.05
20.46
18.32
21.95
40.40
27.75
28.85
28.51
31.38
Nova-6.7B
48.78
30.58
30.85
27.23
34.36
57.47
47.45
43.03
39.68
46.91
Overall, Nova's 's Pass@1 I and Pass@ 10 are higher than all SOTA binary decompilation techniques and general L.MS with even smaller model sizes. Specifically, for each optimization level, Nova consistently decompiles more assemblies into source code correctly than the rest of the compared techniques. Note that Meta LLMCompiler is mainly designed for LLVM IR optimization, and thus is still incapable of assembly code decompilation.
With the same model size, Nova-1.31 outperforms LLM4Decompile-I... 1.3B with a 14.84% higher averaged Pass @ I, and 21.58% higher Pass @ 10. Nova-6.77 7B outperforms L.M44eccmpiie-6.77 with a 12.41% higher averaged Pass 1, and 15.53% higher Pass 10. When compared with GPT-4o, an order of magnitude larger model, Nova- produces an 8.38% higher Pass I and 13.80% higher Pass@ 10. Examples of Nova's correct decompilation are provided in Appendix A.5
Comparison with Techniques Handling Long Input: Nova's 's hierarchical attention design targets to address the low information density and long input challenge of assembly code. There are other techniques that handles long input challenges in text and source code, with Granite-3B--oode -asee----- 128K and LongCoder being the most related ones.
Granite trains L.MM on repository-level long inputs, which is an orthogonal approach with Nova's approach (hierarchical attention and contrastive Icarning). We train Cranite-3B-Code-D1e with Nova's approach, and Table shows that Nova's approach brings improvement to Granite over standard fine-tuning even if has already been trained with long code data.
LongCoder combines window attention and global attention to learn long code input. We compare LongCoder's 'ss attention design with Nova's 's hierarchical attention design by replacing the hierarchical attention of Nova- with LongCoder's attention design. Table Table showss that Nova's 's hierarchical attention is more effective in learning assembly code. Nova's attention design considers the instruction-
8

---

Published as a conference paper at ICLR 2025
Table 3: Nova's approach brings improvement LLM that has been trained with long input data.
Pass@1 I
Pass@ 10
Techniques
Avg.
Avg.
Gramite (3B Code-128K)
5.91
3.78
5.09
5,52
5.08
8.19
5.16
6.56
7.15
6.76
Granite Standard Fine- Tuning
20.88
13.54
11.37
10.09
13.97
30,05
19.77
18.31
15.77
20.98
Granite Nova's 's Approaches
31.04
14.57
14,70
13:66
18.49
39.57
21.23
21.77
19.82
25.60
Table 4: Nova's hierarchical attention is more effective on assembly code.
Pass@
Pass 10
Techniques
00
01
02
03
Avg.
0
01
02
03
Avg.
Nova-1.3B (using LongCoder's Attention)
34.59
19.07
19.72
17.34
22.68
42.19
32.37
32.86
29.04
34.12
Nova-1.3B
37.53
21.71
22.68
18.75
25.17
49.38
34.84
36.95
32.03
38.30
level local semantics and dependencies between different instructions, which fits better than fix-sized window attention to assembly code.
A Ablation Study: We conduct an ablation study by comparing Nova- 1.3B with the following models to show the effectiveness of contrastive learning objectives and hierarchical attention design:
Nova-CL-HA: Removing contrastive learning and hierarchical self-attention. This is simply training DeepSeekCoder-1.33 on the assembly corpus using language modeling. This can be viewed as our reproduction (retrain) of LLM4Decompile-- 1.3B using the same amount of data. AA Removing the hierarchical self-attention, training DeepSeekCoder- 1.3B on the assem- bly corpus using both the language modeling and contrastive learning objectives.
Table 5) shows the results of the ablation study. Nova-CL-HA produces an average Pass@] of 15.78% % and Pass@ 10 of 27.41%. With additional contrastive learning objectives, Nova-HA improves the Pass @1 1 on all optimization levels over Nova Nova-CL-AR HL- showing higher averaged Pass 1 and Pass @ 10. Further applying the hierarchical self-attention on Nova HA boosts the overall Pass from 21.86% to 25.17%, and Pass @ 10 from 35.25% to 38.30%.
5.2 BINARY CODE SIMILARITY DETECTION
Tables 6
8 Sand 9 show the Recall@ I of Nova and existing BCSD techniques with pool size K of 50, 100, 200 and 500 on the six benchmarks. Underline indicates the best in each benchmark, while wave denotes the tied best (we only mark Nova-1.3B for clearer illustration).
7
Overall, Tables 6 7 S and show that on average, Nova-I.3B and Nova 6.7B achieve the highest Recall @1 1 (in bold) under all four settings of K. Nova-6.78 6.7B further outperforms Nova-1.3B and achieves the highest averaged Recall@1 under all four settings, ranking the ground-truth of 5%, 2%, 4%, and 3% consistently outperforms existing techniques with higher Recall when K is 50, 100, and 200, meaning it correctly ranks ground-truth of 3%, 1%, and 2% more queries as the most similar. Under the setting of K 500, Nova-1.33 ties with CodeArt with the same highest Recall 1. When looking into each individual benchmark, Nova- 1.3B always wins on the most benchmarks under different settings of pool size K. For instance, Nova-1.3B wins on four benchmarks while DiEmph only wins on SQLite when K 50. We also conduct ablation study on BCSD in Appendix A.6
queries the most similar correspondingly compared to CodeArt. Nova-1.3BB
5.3 ANALYTIC EXPERIMENTS: How ARE NOVA' 'S EMBEDDINGS BETTER?
We use the widely-used 1-SNE (van der Maaten & Hinton 2008) to analyze and visualize high- dimensional embeddings. We randomly sample seven coding problems from HumenEval-Decon (task_id 19, 32, 34, 63, 119, 128, 143), encode the 00 03 assemblies by Nova- H.AAA and Nova- 1.3B. Figure 5 shows the embeddings that are visualized under the first principal components. Each color represents one task, and 09 03 assemblies are marked by O. V. A, and D.
Compared with Nova-CL-HA (Figure 5()r)) (a)). Nova- (Figure 5 (b)) including contrastive learning objectives in the pre-training, can separate the embeddings of assemblies with different functionalities better. Nova A clearly encode "Task 143" (orange points) away from the others. Yet, Nova's

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study Nova-1.3B on HumanEval--ecompile
Pass@1
Pass 10
Techniques
Avg.
Avg.
LLM-Decompile 1.3B
15.30
8.26
9.36
8.38
10.33
21.79
15.23
16.17
13.70
16.72
Nova CL
20.73
16.16
15.03
11.19
15.78
33.55
28.12
26.96
21.01
27.41
Nova NA
20.58
16.40
44.75
Nova
37.53
21.71
22.68
18.75
25.17
49.38
34,84
36.95
32.03
38.30
Table 6: Recall@1 1 on BCSD with K 50
Table 7: Recall @ on BCSD with K 100
Benchmarks
Trans
DiEmph
CodeArt
Nova-6.7P
Benchmarks
|iTrans
DiEmph
CodeAnt
Nova-1.31 [Nova-6.7B
Binails
0.68
0.80
0.84
0.87
0.89
Binutils
0.60
0.63
0.81
0.79
0.79
Curl
0.72
0.84
0.86
0.89
0.94
ImageMagick
0.53
0.71
0.78
0.86
0.90
ImageMagick
0.54
0.71
0.76
0.79
0.81
SQLite
SQLite
0.62
0.72
0.74
0.73
OpenSSL
0.70
0.83
0.88
0.90
0.92
OpenSSL
0.60
0.80
0.87
0.88
0.90
Putty
0.63
0,72
0.69
0.72
0.71
Putty
0.58
0.64
0.64
0.65
Avg.
0.67
0.78
0.81
0.84
0.86
Avg.
0.60
0.72
0.77
0.78
0.79
Table 8: Recall @1 on BCSD with K
200
Table 9: Recall on BCSD with K = 500
Benchmarks
Trans
DifEmph
CodeArt
Nova: 1.3B
Nova 6.7B
Benchmarks
|irrans
DiEmph
CodcArt Nova-1.3B Nova-6 7B
Binuiiss
Binatils
0.65
Curl
0.57
0.77
0.78
0.83
0.84
Curl
0.43
0.62
0.69
0.73
0.76
ImageMagick
0.39
0.51
0.75
TmageMagick
0.25
0.42
0.58
0.61
0.65
SQLite
0.56
0.65
0,68
0.68
0.69
SQLite
0.59
0.62
0.59
0.62
OpenSSL
0.54
0,71
0.82
0.84
0.88
OpenSSL
0.43
0.61
0.76
0.78
0.82
Putty
0.49
0.58
0.55
0.58
Putly
0.47
Avg.
0.51
0.64
0.71
0.73
0.75
Avg.
0.39
0.55
0.64
0.64
(Figure (c)) embeddings group the assemblies by functionalities more precisely than Nova H.. suggesting that hierarchical attention enhances the training of contrastive learning objectives to learn more effective encoding. Visualization using a different approach, PCA, shown in Appendix A.7 Analytic experiments Nova's hierarchical attention is shown in Appendix A.8
20-
Task 143
20-
Task 19
10-
Tack
10
Task
assersbllcs Mesccaa Bich
-30
-10
Task 32
-60
-20
20
20
20
(a) NOvA. Embeddings
(b) Nova, Embeddings
(0) Nova's Embeddings
Figure t-SNE analysis of embeddings calculated by Nova-CL-HA, Nova DO and Nova.
CONCLUSION
This work presents Nova, a generative foundation LLM for assembly code that introduces two key innovations hierarchical attention and contrastive learning to tackle the unique challenges of assembly code understanding. Nova builds upon a source code L.LM and undergoes further pre- training approximately 4.3 million collected assembly functions. We fine-tune and evaluate Nova on two downstream tasks: binary code decompilation and binary code similarity detection. Nova achieves to 14.84 21.58% (absolute percentage point improvement) higher Pass @ I and Pass @ 10 over existing methods in binary code decompilation and up to a 17% higher Recall @ in binary code similarity detection. Looking forward, believe that the proposed hierarchical attention and contrastive learning methods hold promise for enhancing foundation models both source code and natural language domains, which leave future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research was supported in part by NSF 1901242 and 2006688, a CFI fund, and J.P. Morgan AI Faculty Research Awards. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
REFERENCES
Simone Aonzo, Yufei Han, Alessandro Mantovani, and Davide Balzarotti. Humans vs. Machines in Malware Classification. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1145-1162, Anaheim, CA, 2023. USENIX Association. ISBN 978- 1-939133--77--3 URL
aonzo
Jordi Armengol-Estapé,, Jackson Woodruff, Chris Cummins, and Michael F. P. O' O'Boyle. SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Kerbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models Few-Shot Learners. CoRR, abs/2005. 14165, 2020. URL httpht////axxxx http:::///axxi.ooo///bbbbb/
14165
Martim Carbone, Weidong Cui, Long Lu, Wenke Lee, Marcus Peinado, and Xuxian Jiang. Mapping kernel objects to enable systematic integrity checking. In Proceedings of the 16th ACM Conference Computer and Communications Security, CCS '09, pp. 555-565, New York, NY, USA. 2009. Association for Computing Machinery. ISBN 9781605588940. doi: 10.144//663662..16377..
URL http:httt://diiooot http///oo.orr///////// 10. 1145/1653662 1653729
Nicholas Carlini, Antonio Barresi, Mathias Payer, David Wagner, and Thomas R. Gross. Control- On the Effectiveness of Control-Flow Integrity. In 24th USENIX Security Sym- posium (USENIX Security 15), pp. 161-176, Washington, D.C., 2015. USENIX Association.
Flow Bending:
ISBN 978-1-939133--1-- URL htw.h: http://:////wwuuueii... www.ux.ee...............
Lei Chai and Ming Li. Pyramid attention for source code summarization. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 978171387 1088.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Improving Code Generation by Training with Natural Language Feedback, 2023.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Guang Lou, and Weizhu Chen. CodeT: Code Generation with Generated Tests, 2022a. URL 10397
Ligeng Chen, Zhongling Hc, Hao Wu, Fengyuan Xu, Yi Qian, and Bing Mao. DIComP: Lightweight Data-Driven Inference of Binary Compiler Provenance with High Accuracy. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 112- 122, 2022b. doi: 10. 1011199ANNE53333222200000
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa,
11

---

Published as a conference paper at ICLR 2025
Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained Code. CoRR, abs/2107.03374, 2021. URL https://arxiv org/abs/2107 03374
Qibin Chen, Jeremy Lacomis, Edward J. Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Augmenting Decompiler Output with Learned Variable Names and Types. 3/st USENIX Security Symposium (USENIX Security 22), pp. wwwp:/wwwwwwwwwptttptpptwwttt.wp www.wx.eeii.... wrw./oneeeccocccce////
Boston, MA, aug 2022c.
USENIX Association. ISBN 978-1-939133-31-11 URL https:: /wwp://wwww.cco
Simin Chen, Hamed Khanpour, Cong Liu, and Wei Yang. Learning to Reverse DNNs from AI 10025/2222221064444
Programs Automatically, 2022d. URL
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT's Attention. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACI. Workshop BlackboxNI.P Analyzing and Interpreting Neural Networks for NLP. pp. 276-286, Florence, Italy, August 2019, Association for Computational Linguistics. doi: 10.18653/v 1/W 19-4828. URL httpss//aalaaannhooooo org/
W19-4828
Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta Large Language Model Compiler: Foundation Models of Compiler Optimization, 2024. URL htttp:://arxii.ogg/aab////
.25244
Anderson Faustino da Silva, Bruno Conde Kind, José Wesley de Souza Magalhães, Jerônimo Nunes Rocha, Breno Campos Ferreira Guimarães, and Fernando Magno Quinão Pereira. ANGHABENCH: A Suite with One Million Compilable Benchmarks for Code-Size Reduction. In 2021 IEEE/ACM International Symposium Code Ceneration and Optimization (CGO), pp. 378-390, 2021. doi: 10. N0.1109//051591.00 5902 2021.990222....
Yaniv David and Eran Yahav. Tracelet-Based Code Search in Executables. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 14, pp. 349-360, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 978 1450327848. doi: 10.1145/2594291.222 25993333. URL https:tttt/.oooooo 14444/15550///1111144/444444 1145/2594211 2594343
Yaniv David, Nimrod Partush, and Eran Yahav. Statistical Similarity of Binaries. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '16. pp. 266-280, New York. NY, USA, 2016. Association for Computing Machinery. ISBN 978 150342212.. doi: 10. 145/2908880029008
2968126
Yaniv David, Nimrod Partush, and Eran Yahav. Similarity of Binaries through Re-Optimization.
SIGPLAN Not., 52(6): 94, jun 2017. ISSN 1362-1340. doi: 10.1145731405577306633777 URL
https://doi org 10455...1111 0145/31445877 30623877
Yaniv David, Nimrod Partush, and Eran Yahav. FirmUp: Precise Static Detection of Common
Vulnerabilities in Firmware. SIGPLAN Not., 53(2):392 404, mar 2018. ISSN 0362- 1340. doi: 10.1145/329995.337771 URL
3177117
Steven H. H. Ding, Benjamin C. M. Fung, and Philippe Charland. Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 472-489, 2019. doi:
10.1109/SP2211.000000
Yue Duan, Xuezixiang Li, Jinghan Wang, and Heng Yin. DeefBinDiff: Learning Program Wide Code Representations for Binary Diffing. 01 2020. doi: 10.14722hnss.2222..5511
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A Generative Model for Code Infilling and Synthesis, 2023.
12

---

Published as a conference paper at ICLR 2025
Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushan- far, and Jishen Zhao. Coda: An End-to-End Neural Program Decompiler. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Bue, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 440006ee85584440baacf44443422222
neuriss. wwtpppppr..llls/ppppeeeeeeeeeeee
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, Pp. 6894 6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.16653ww120021.cmlm--mmo
URL https
//aclanthology.org org org/ emnlp-man -o0mmmin.... 552
Emre Güler, Comeelis Aschermann, Ali Abbasi, and Thorsten Holz. AntiFuzz: Impeding Fuzzing Audits of Binary Executables. In 28th USENIX Security Symposium (USENIX Security 19), pp. 1931-1947, Santa Clara, CA, 2019. USENIX Association. ISBN 978-1-9391333-0-4 URL
https://www. usenix
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. LongCoder: a long-range pre- trained language model for code completion. In Proceedings of the 40th International Conference Machine Learning, ICML'23. JMLR.org, 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. SeepSeek-Coden When the Large Language Model Meets Programming The Rise of Code Intelligence, 2024.
Jingxuan He and Martin Vechev. Large Language Models for Code: Security Hardening and Adversarial Testing. CORR, 105/23022.03119, 2023.
Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin Vechev. Debin: Predicting
Debug Information in Stripped Binaries. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS '18, pp. 1667-1680, New York. NY, USA, 2018.
Association for Computing Machinery. ISBN 978 1188136656300 doi: 10.1 00.1145/2243734.32436666
URL https: orooorg//// 10. 01145/2443344444 3243866
Iman Hosseini and Brendan Dolan-Gavitt. Beyond the C: Retargetable Decompilation using Neural Machine Translation. In Proceedings 2022 Workshop on Binury Analysis Research, BAR 2022. Internet Society, 2022. doi: 10.14.2220 URL phttp://dk doi org 14722/ bar. 2022..30009
Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, and Penghao Zhao. Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey, 2024. httpshtss///arii org/abs/2311.cm 12351
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-Coder Technical Report. arXiv preprint arXiv:2409. /2/86, 2024.
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of Code Language Models on Automated Program Repair. In Proceedings of the 45th International Conference Software Engineeringg ICSE '23, pp. 1430- 1442. IEEE Press, 2023. ISBN 9781 665457019. doi: 10.1 ICSE48619.2023.009 00.22.. URL https::///ot wrw.orgg///// 10111/999/111119991 2023.00125
Jeremy Lacomis, Pengcheng Yin, Edward J. Schwartz, Miltiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. DIRE: a neural approach to decompiled identifier naming. In Proceedings of the 34th IEEE/ACM International Conference Automated Software Engineering, ASE 19, pp. 628 639, IEEE Press, 2020. ISBN 9781728 0181728125084, doi: 10. 10.11000 109/AAS..00 20000009000000 19.000064.
URL https: doi. org 10. 1109/ASE. 2019. 00064
13

---

Published as a conference paper at ICLR 2025
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. arXiv preprint, 1ww.22077017800 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-tor Leaeeencett-Sequeee to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, 2019.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhski,, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze. Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov. Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you!, 2023.
Ruigang Liang, Ying Cao. Peiwei Hu, and Kai Chen. Neutron: attention- neural decompiler. Cybersecurity, 4(1); 2021. ISSN 2523-3246. doi: 10.1 0011186/42400000-00000000
URL https
//doi.org/1e, 11
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: Reinforcement Learning from Unit Test Feedback, 2023a.
Zhibo Liu, Yuanyuan Yuan, Shuai Wang, Xiaofei Xie, and Lei Ma. Decompiling x86 deep neural network executables. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC '23, USA, 2023b. USENIX Association. ISBN 978- 778-1-939133-37-3.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary. Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder 2 and The Stack V2: The Next Generation, 2024. URL http/ht/s////xii org/ 2402. 19173
Lannan Luo, Jiang Ming, Dinghao Wu, Peng Liu, and Sencun Zhu. Semantics- -Based Obfuscation- Resilient Binary Code Similarity Comparison with Applications to Software Plagiarism Detection. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pp. 389-400, New York, NY, USA. 2014. Association for Computing Machinery, ISBN 9781498 4503335055 1145/2635868 2635900
doi:
URL https:///oi. org/10
Jean Phinllpilipe Martin, Michael Hicks, Manuel Costa, Periklis Akritidis, and Miguel Castro. Dy- namically checking ownership policies in concurrent c/c++ programs. SIGPLAN Not., 45 (1):4574470, jan 2010. ISSN 0362-1340. doi: 10. 77.114//77788017776311 URL httpss
//doi.org/1e. 1145/777781 1706351
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan- Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, Yan Koyfman, Boris Lublinsky, Maximilien de Bayser, Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Yi Zhou, Chris Johnson, Aanchal Goyal, Hima Patel, Yousaf
14

---

Published as a conference paper at ICLR 2025
Shah, Petros Zerfos, Heiko Ludwig, Asim Munawar, Maxwell Crouse, Pavan Kapanipathi. Shweta Salaria, Bob Calio, Sophia Wen, Seetharami Seelam, Brian Belgodere, Carlos Fonseca, Amith Singhee, Nirmit Desai, David D. Cox, Ruchir Puri, and Rameswar Panda. Granite Code Models: A Family of Open Foundation Models for Code Intelligence, 2024. httlhtttp:///rxii..oo abs/2405. 04324
Minh Huynh Nguyen, Nghi D. Q. Bui, Truong Son Hy. Long Than,, and Tien N. Nguyen. HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representations, 2023.
URL htt.ttttp///aiiiii www.aaawww.....raaaaaaaaeerareee wtt.orrg//bb/////// 15479
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A Conversational Paradigm for Program Synthesis. arXiv preprint, 2022.
OpenAl. GPT-4 Technical Report, 2023.
Kexin Pei, Zhou Xuan, Junfeng Yang. Suman Jana, and Baishakhi Ray. Trex: Learning execution semantics from micro-traces for binary similarity. arXiv preprint arXiv:2012.0866 2020.
Kexin Pei, Jonas Guan, Matthew Broughton, Zhongtian Chen, Songchen Yao, David Williams-King, Vikas Ummadisetty, Junfeng Yang. Baishakhi Ray, and Suman Jana. StateFormer: Fine-grained type recovery from binaries using generative state modeling, In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 690-702, 2021.
Alec Radford and Karthik Narasimhan. Improving Language Understanding by Generative Pre- Training. 2018. URL https: wew.eeitsaattnttoooeeeeeo org/CorppsID:4931133
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open Foundation Models for Code, 2023.
Andreas Sæbjørnsen, Jeremiah Willcock, Thomas Panas, Daniel Quinlan, and Zhendong Su. Detect- ing Code Clones Binary Executables. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA '09, pp. 117-128, New York, NY, USA, 2009. Associa- tion for Computing Machinery. ISBN 978 6055883389 doi: 10.114511572272.5572277 URL httpp:://oi org/10.. /0. 1045/ 157 2272.222..11 1572287
Calif. Sausalito. Cybercrime Bytes: 10 Hot Security Certs, Public Safety Hacked, Intru- sion's Shield. Cybercrime Magazine, 2020. URL http:p::://cccbeeeeeerrttttetrss
Subercrime-bytes hrt.a//ree...tttttt...tooooooaen
Ensheng Shi, Yanlin Wang. Lun Du, Hongyu Zhang. Shi Han, Dongmei Zhang, and Hongbin Sun. CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 4053-4062, Online and Punta Cana, Dominican Republic, November 2021. URLhtttss
Association for Computational Linguistics. doi: 1211665//1/2221.emmmmmmit 332
//aclantholoog.org// wwwp:/loooyoo.ggg2g0 2021 wttp://2221ee.e111mmeemm
Chad Spensky, Hongyi Hu, and Kevin Leach. LO-PHI: Low-Observable Physical Host Instrumenta- tion for Malware Analysis. 2016.
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. An Empirical Study of Deep Learning Models for Vulnerability Detection. In 2023 IEFF/ACM 45th International Conference on Software Engineering (ICSE), pp. 2237-2248, 2023. doi: 11L.119//SEA4499.220000
Zian Su, Xiangzhe Xu, Ziyang Huang, Zhuo Zhang. Yapeng Ye, Jianjun Huang, and Xiangyu Zhang. CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking, 2024.
15

---

0001009/CSEEA66122223300000 Published as a conference paper at ICLR 2025
Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. LLM4Decompile: Decompiling Binary Code with Large Language Models, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 13971, 2023.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579-2666 2008. URL http://jmlr org/
vandermaaten@&a.hhr htmm
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https:::/procceeeigg... wwwp:/ss.w........pppp.e.pe.. 3 faaos///eeee1fffsscccccccc4aaaaa
pdf
Hao Wang, Wenjie Qu, Gilad Katz, Wenyu Zhu, Zeyu Gao, Han Qiu, Jianwei Zhuge, and Chao Zhang. JTrans: Jump-Awar Aware Transformer for Binary Code Similarity Detection. In Proceedings of the
st ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2022, pp. 1-13, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450333799 doi: 00.1145/3533767.355.3
utthhtps:: //doi 110 1145/353376/ 3534367
Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. How Effective Are Neural Networks for Fixing Security Vulnerabilities. In Proceedings the 32nd ACM SIGSOFT International Symposium Software Testing and Analysis, ISSTA 2023, pp. 1282-1294, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702211. 1. doi: 10. 10.1145/3599966.59988 35917999
URL https://6oi. 1145/
3597926 3598135
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated Program Repair in the Era of Large Pre- Pre-Trained Language Models. In Proceedings of the 45th International Conference on Software Engineering, ICSE '23, pp. 1482-1494. IEEE Press, 2023, ISBN 978 665457019. doi: HRLLhttps://doi orgo 10. 1109/ICSE482 19. 2023. 00023.00222 00129
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations, 2024. URL /tt/ttts////ppeeeeiiiee
Xiangzhe Xu, Shiwei Feng. Yapeng Yc, Guangyu Shen, Zian Su, Siyuan Cheng, Guanhong Tao, Qingkai Shi, Zhuo Zhang, and Xiangyu Zhang. Improving Binary Code Similarity Transformer Models by Semantics Driven Instruction Deemphasis. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, pp. 1106-1118, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400702211. 10.1145/2597926.55988
Dervnnics-Driven
doi:
URL httth::/dddi.oorr
Xiangzhe Xu, Zhuo Zhang. Shiwei Feng, Yapeng Ye, Zian Su, Nan Jiang, Siyuan Cheng, Lin Tan, and Xiangyu Zhang, LmPa: Improving Decompilation by Synergy of Large Language Model and Program Analysis, 2023b.
Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS '17, pp. 363-376, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 450039466. doi: 10.1145/3139956.114401 URL httpt:://tot.orrg/ 114513..111111111111 3134018
Zhaoyan Xu, Antonio Nappa, Robert Baykov, Guangliang Yang. Juan Caballero, and Guofei Gu. AUTOPROBE: Towards Automatic Active Malicious Server Probing Using Dynamic Binary Analysis. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, CCS '4, pp. 179-190, New York, NY, USA, 2014, Association for Computing Machinery.
16

---

Published as a conference paper at ICLR 2025
ISBN 9781450329576. 2668267. 2560352
doi: 10. 10.1145/266026722666333. URL
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical Attention Networks for Document Classification. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480-1489, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/ N16-11744
1174
Wang Yue, Wang Weishi, Joty Shafiq, and C.H. Hoi Steven. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decode Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, 2021.
Zhuo Zhang, Yapeng Ye, Wei You, Guanhong Tao, Wen-chuan Lee, Yonghwi Kwon, Yousra Aafer, and Xiangyu Zhang, OSPREY: Recovery of Variable and Data Structure via Probabilistic Analysis for Stripped Binary. In 202/ IEEE Symposiun Security and Privacy (SP), pp. 813-832, 2021. doi: 10. 10.1109/SP4000..0 2021000051.
A APPENDIX
A.1 DATA COLLECTION
This section provides additional details of the data collection. To collect assemblies from The-Stack, we attempt compile 4 million programs, of which 138.8K is compiled successfully. We do not collect more due to the computation resource limitations.
For the 757.1K and 138.8K source code that successfully compiled into executables (using gcc) from AnghaBench and The-Stack, we disassemble them using objdump. objdump was not able to successfully disassemble all the executables, resulting some empty assembly code. Thus, the number of 0e 01 we obtain from each corpus is different and smaller than the number of source codes as shown in Table
There may be alternative ways of collecting assembly code, e.g., using gee -S to directly dump the assembly code without producing executables. However, key difference is that the assembly generated by gec -S does not undergo the linking step. In practice, binary decompilation and analysis are typically performed on executable or linked assembly code, as it includes linker modifications and reflects the final binary structure. Our data collection aligns better with the practical use of assembly. Nevertheless, Nova's approaches should be generalizable to assembly code obtained with a different approach, and the experiments remain future work.
Figure shows an example of preprocessing the raw assembly code as described in Section 3.1
0:
endbr64 push
endbr64
[iNST-0]
krbp
push rbp mov rsp rbp
INST-1
5:
mov
Srspp,rrp
INST 2
88:
mov
Krdi, -8x8(
mov rdi
-8 ( rbp )
[INST-3 31
<::
Krsi, -8x1 -0xx0xxxxp)))
-16
INST-44 47
mov
-0x8(Xrbp) krax
(rrp
( rbp ) rax
14:
mov
(Srax) Sedx
mov
rax ) edx
INST-6 61
16:
mov
Srax Srax (Rrax) Seax
@x10(krbp).
mov -16 ( rbp
rax
INST-7
mov
( rax )
INST-87
1c:
cmp
Reax, Sedx
cmp
eax edx
[INST-9
le:.............................
(a) Raw Assembly
(b) Normalized Assembly
Figure 6: Example of assembly code preprocessing
A.2 EXPERIMENTAL SETUP DETAILS
This section provides additional details of training. We pre-train Nova starting from SeepSeck-Codee using the language modeling objective (Li.. for one epoch on the functions and assembly functions collected from AnghaBench and The-Stack corpora. This step uses batch size of 128, with the input
17

---

Published as a conference paper at ICLR 2025
Table 10: Comparison with applying hierarchical attention on all attention heads, using IB models.
Pass@1 I
Pass 10
Techniques
Avg.
01
Avg.
Nova A
30.58
19.88
20.58
16.40
21.86
44.75
33.13
33.31
29.82
35.25
Nova (hierarchical all heads)
32.38
18.87
20.56
16.34
22.04
45,95
32.19
32.78
29.01
34.98
Nova
37,53
21.71
22.68
18.75
25.17
49.38
34.84
36.95
32.03
38,30
truncated by a 1,024 tokens limit. The model weights are updated using the AdamW optimizer. The learning rate is 5e 5, using 1000 steps of warm-up and a cosine decay to adjust the learning rate.
Then, the model further pre-trained with the combination of language modeling and contrastive learning objectives (L Lim. + X(L + Loet), with À set to 0.1. To train with the functionality contrastive learning objective, we discard any source code that misses any one of 00 03 assemblies and also discard the source code whose 02 assembly is the same its assembly. As a result, this step is only trained for 0.36M data samples for one epoch. The batch size 64, with the input truncated by a 1,024 tokens limit. The learning rate 2e using the AdamW optimizer.
The fine-tuning of both BCD and BCSD uses a batch size of 64, with the input truncated by a 2,048 token limit. Similarly, the learning rate is 2e-5 using the AdamW optimizer, and the model is fine-tuned for one epoch. During the training using the contrastive learning objectives, and the fine-tuning of BCSD, we use the average of 
 tokens' last layer hidden states represent the embedding of a binary function.
For the evaluation of BCSD, we reuse the framework provided by CodeArt (Su et al. (2024) to evaluate the binary code similarity detection results once Nova produces the embeddings for functions in the test dataset. For each one of the K functions, the assembly function is used as the query to calculate the cosine similarity between its embedding and the embeddings of all the K 03 assembly functions. Note that Nova does not normalize the embeddings of assembly functions during training and uses l2 distance to calculate the LBCSD. which optimize Nova to embed the 00 and 03 assembly functions from the same source code have the smallest /2 distances. When using CodeArt's 's framework for evaluation which ranks the similarity using the cosine similarity, normalize Nova's embedding for each assembly function since l2 distance after normalization keeps the same order as cosine similarity (smaller l2 distance means higher cosine similarity).
Infrastructure The experiments are conducted on eight NVIDIA RTX A100 GPUs, each with 40GB memory, Our implementation is based on Huggingface's implementation of DeepSeck-Coder PyTorch and DeepSpeed
A.3 APPLYING HIERARCHICAL ATTENTION ON HALF ATTENTION HEADS
The hierarchical attention mask is always applied on half of the attention heads at each layer in Nova. This ensures the LLM balances the hierarchical knowledge of assembly code and pre-trained knowledge learned by full self-attention.
We conducted experiments applying hierarchical attention to all the attention heads. Results in Table show that when applying hierarchical attention to all the attention heads of transformer layers, the performance does not improve enough and even drops under some settings. This implies that the standard full self-attention mechanism indeed learns knowledge that may not be captured by hierarchical attention. Thus, we only apply hierarchical attention to half of the attention heads in each transformer layer of Nova to balance the knowledge learned by standard and hierarchical attention.
A.4 ADDITIONAL ABLATION STUDY ON BINARY CODE DECOMPILATION
We provide additional ablation studies studying the impact of each individual contrastive learning objective. We study two more models:
Nova-rcL-Ha: Removing functional contrastive learning and hierarchical self-attention.
babb
18

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study of Nova-1.3B on HumanEval-Decompe
Pass @
Pass 10
Techniques
Avg.
Avg.
Nova CL RA
20.73
16.16
15.03
11.19
15.78
33.55
28.12
26.96
21.01
27.41
Nova FCL HA
22.38
16.20
16.37
13.25
17.05
36.13
29.48
30.02
23.76
29.85
Nova OCL- HA
28.44
18.87
18.53
15.76
20.40
40.28
32.33
31.80
27.05
32.87
Nova-NA WA
20.58
16.40
3..66
44.75
33.13
33.31
29.82
35.25
Nova
37.53
21.71
22.68
18.75
25.17
49.38
34,84
36.95
32.03
38.30
Nova-OCL-HA. Removing optimization contrastive learning and hierarchical self-attention.
Table 11 shows that each component of the contrastive learning brings certain improvements to the Pass @ and Pass @ 10 on BrammanEval-Deoompll and we find the impact of function contrastive learning (FCL) is larger than the impact of optimization contrastive learning (OCL), suggesting that aligning the model's embeddings for assembly code with the same functionality is more useful.
kfunce>
Lin Dattt farr, int n) for (int 1 1; < i+t) for (int - ii; ir [[abs(ar:[[] aFR 1.00)
8: endbr64
test
66: ]Le
48 <tunc0+0x48>
88: lea
-xx((Srs)). Secx
return 1;
:E: nov
$ax1, Kedx
114:
exe($rip), 3xmn3
} return 0:
1b: 00
GPT-4's Incorrect Decompilation
3e:
Brex, Andx
:1: je
<func0+0x4e>
21: novss
-xx((krdt Andx, 4), Exmm?
27
Srdx,Srax Srax
int funco(const. float *x, int n) { int 1; 1.0F;
2za: novaps SxnnZ, sxnet
2d. subss
(Srdi. $zax. 4), 3xmm1
:32 andps
Sxnn3, Sxnmi
for (1 1: 1 itt)
335: coniss
Sxnni, Nxmne
(tabs(x[1] x[1-1])
338 ja
54 <func2+8x5d>
return return 1; LLM4Decompil'ss
S0x1,
3e: cmp
Seax, tesi
Decompilation
48: 1g
Za <tunc0H0x2a>
$0x1, ic <func@+ekie>
446 jep
funco(float *a, float c) int i.
14:
$8x0, Neax
4dd reto
for (i 8: <; i+t)
4e: nov
$2x8, Neax
(1 < f+4)
(fabs(a[i] a[5])
54:
59: retg
nov
$0x1, Neax
return 1:
return
Nova's Correct
01 Assembly Code
Decompilation
Figure 7: Nova-1.3B correctly decompiles HumanFiyal-Decomme task O.
A.5 BINARY CODE DECOMPILATION CASE STUDIES
Figure 7 shows example from HumanEval Eummmmill-Decompple (task id 0). Given the 01-optimized assembly code, GPT-4 fails to figure out the number of function arguments correctly, missing one important argument "float e", and thus produces wrong functionality in the decompiled code. LLM44ccompiie-1.33 makes similar mistakes and also misses the inner nested for loop. Nova-1.3B correctly decompiles the assembly into source code, where the ground truth is checking if any two elements in the given list *a (with size n) are close to each other than given threshold
Figure X shows another more complex example, HumanEval- Dlucompyal- task-id 20. Nova- 1.3B correctly decompiles the source code, successfully figuring that the function is trying to find the two elements that are closest to each other in the given array *a, with the result stored in minmax.
A.6
BINARY CODE SIMILARITY DETECTION ABLATION STUDY
Table 12
1
44
15
show the detailed ablation study results of BCSD. Nova wins on the most 100 500, and ties with Nova when K 50, or 200.
benchmarks when K
19

---

Published as a conference paper at ICLR 2025
xfunce>:
void Iunce(Iloat *a, int n. float *b)
8:
endbr64
int i, 3;
(Rzdi),
max:
8: movss
Nxmm8, (Ardx)
0x4(3rdi), Nxnr1 0xd(Xrdx)
min (i a[e]:
movss
i++) {
test
Nes1, Nest
1f [a[1] < nim)
:8:: i
8b <funcBB6x8h>>
min a[i];
11a:
Lea
-xx((srs)) 8r9d
etse [a[3] max]
:1:: nov
sax1, Kröd
nax a[3]]
224
mov
Srdi. Rrox @xe(krip), Xxmm
227
movss
b min: b[1]
2ee
LLM4Deommpil''s Incorrect Decompilation
2E:
355:
#xe(krip), 3xmm movss
3aa:
33h
cmp kr8, Kr9
woid tunce(float *2, n, float Mminnaxx { mirmax[0] a[o]: mirmax[1] a[1];
3e:
82 <func0+8x82>
48:
mov
nopt #x0[$rax, trax, 1)
488
movss (Rrox), Sxmn1
movaps Sann1,
(int for (int - 1 ist)
441
subss (Rrd1, krax, 4), Rorme
(++)
154:
andps Sxmn3, Exmm0
float diff [abs(a[A] a[i)):
Samn, jbe 6d <func0+0x6d>
minmax[1] mirmax(9])
Saa:
Se:
Sxmn1, (3rdx)
minnax[o] a[i]
66:
movss (Krdi, $rax, 4), 8xmm1
minnax[2] a[il:
665
novaps Samne, 1xmm2
668
Nxmni, ex4(Xrdx)
6d:
s8x1, Krax Seax, Sest cmp
add
71:
73:
3g
<tuncoH0x48>
[ninmax[0] minmax[1]] float temp minmax[R];
75:
add
$8x1, Kr88
79:
add
$0x4, Krex
ninnax[0] minnax[1]: minnax[1] teep:
99:
zetq
Nova's Correct
02 Assembly Code
Decompilation
Figure Nova-1.3B correctly decompiles HumanFyal-Decomppl task 20.
Table 12: Ablation study with K = 50.
Table 13: Ablation study with K 100.
Benehmarks
Nova CL- HA
Nova HA
Nova-1.33
Benchmarks
Nova
Nova
Nova 1.3B
Binurils
Binutils
0.80
0.82
0.79
Curl
0.84
0.87
0.89
Curl
0.84
0.84
0.86
ImageMagick
0,79
0.80
0.86
ImageMagick
0.70
0.72
0.79
SQLite
0.80
0.83
0.77
SQLite
0.74
0.78
0.73
OpenSSL
0.90
0.92
0.90
OpenSSI.
0,89
Putty
Putty
0.59
Avg.
0.83
0.84
Avg.
0.76
0.28
0,78
Table 14: Ablation study with K 200.
Table 15: Ablation study with K
500.
Benchmarks
Nova CL- HA
Nova
Nova:
Benchmarks
Nova Nova-CL-HA
Nova HA
Nova-
Binutils
0.71
0.74
0.73
Binutils
0.62
0.65
0.65
Curl
0.67
0.71
ImageMagick
0.61
0.63
0.73
ImageMagick
0.46
0.51
0.61
SQLite
0.71
0.68
SQLite
0.61
0.62
0.59
OpenSSI.
0.85
0.87
0.84
OpenSSL
0.77
Putty
0.53
0.53
0.56
Purty
0.46
0.46
0.47
Avg.
0.70
0.70
0.73
Avg.
0.60
0.62
0.64
20

---

Published as a conference paper at ICLR 2025
0.6
0,5
Task
0,4
Tesi
0.0
Task 34
-0.
-0.5
0.2
0.20
Tisk 128
-0.55
-0.75
-0.50
-0.5
0.00
0.55
-0.30
-0.25
0.00
0.25
30
(a) NOVA,CT.HA Embeddings
(b) Noviuuu, Embeddings
(6) Nova's Embeddings
Figure 9: PCA of embeddings calculated by Nova -L- HA. Nova MA. and Nova.
Nova-ct.-
A.7 ADDITIONAL ANALYSIS OF EMBEDDING
Figure shows the results of PCA of embeddings provided by Nova-ci CL---C--A--- Nova HA, and Nova, on randomly sampled seven examples, where Nova's embeddings are consistently more distinguishable by functionalities.
Th
IIST
(IST
IN5T33
(IST
moy
mov
[NSTTd]
(a) Last Layer, Head 0 (Standard Affention)
(c) Last Layer, Head 10 (Htiearrcicaal Aftention)
(WST22)
(N6544
wuh
INS571
IIST.
push
push
bp
bo
[AST-4
[R6T-4]
(b) Last Layer, Head 4 (Standard Atteation)
(d) Last Layer, Head 14 (Hierarchleal Attention)
Figure 10: Comparison of attention distribution among standard and hierarchical heads.
A.8 ADDITIONAL ANALYSIS OF ATTENTION
Figure shows the visualizations of attention weights the final transformer layer of two select heads with standard attention and two heads with learned hierarchical attention. Standard attention
21

---

Published as a conference paper at ICLR 2025
(NST11
(NST
(INTT-1
noo
e00
I
push
bp
[INST
moy
bp
INNTT
mov
[NNST
First Layer, Head 8 (Hfierarchical Attention)
Figure 11: Learned per-instruction soft attention observed in the lower layers
exhibits two typical patterns, namely diagonal attention (i.e. tokens attending to themselves or nearby tokens, shown in Figure (a)), and broad attention (i.e. single token attending broadly the entire sequence, shown in Figure 10 (b)). In contrast, in Nova's hierarchical attention, attention weights are allocated among distinct segments, each corresponding to instruction (shown in Figure I0 (c)), that focus on tokens comprising that instruction (e.g. opcodes and operands, shown in Figure 10 (d), attentions paid to "push", "mov". etc.).
Quantitatively, we have determined broad attention accounts for as much as 30% all attention in standard heads, especially layers 1-8 (consistent with the findings of (Clark al. 2019)), whereas in Nova's hierarchical attention, no more than 5% or all attention is allocated to each instruction segment. This validates our goal learning instruction-ane aware hierarchical attention Nova.
In addition, in lower layers, we have observed attention weights to be softly distributed among tokens comprising each instruction (Figure [T]. which suggests Nova initially models cross-relations among operation codes and operands in the first few layers, and later pools their summary representation into the 
 token the later layers.
A.9 LIMITATIONS
One limitation is that Nova is X86-specific, we only collect X86 assembly corpus for pre-training.
This design choice mainly affected by two facts: (1) X86 assembly is used and explored
wide range of binary tasks (Wang al.) 2022 Su al., 2024 Xu al. 2023a| Chen et al.) 2022c)
compared to other assembly languages, and (2) computation limitations. However, the proposed techniques independent of X86 assembly. Low information density and compiler optimization the common challenges of most assembly languages such X86, ARM, and MIPS. The proposed techniques can be applied the future development of multi-lingual assembly LLMs. Another potential limitation is the scale of models. We develop Nova-1.3B and Nova-6.7B and demonstrate their advancement in assembly code decompilation and encoding. Developing larger Nova models promising. but remains future work due to limited computing resources.
22


Reviews:
[{"review_id": "88KucOADA5", "reviewer": "Reviewer_MUPt", "summary": "The paper presents a way of training an LLM to improve its performance on tasks that require understanding of assembly code, in particular code decompilation, and assembly code similarity detection.\n\nThis is achieved by several contributions:\n1. A multi-way, parallel corpus of programs written in C, as well as the corresponding assembly produced by `gcc` with different levels of optimization (0 to 3), used for further training of pre-trained LLMs.\n2. A hierarchical attention mechanism, structured to summarize the content of each instruction into the representation of a single token. This mechanism is compatible with existing models.\n3. Two auxiliary contrastive loss objectives: a \"functionality\" one that minimizes the distance between representations of the same original code, while maximizing the distance between representations of different code pieces, and an \"optimization\" one encoding the fact that further levels of optimization should increase the distance between program representations.\n\nTwo variants (with 1B and 6B parameters respectively) of a model trained with these changes, and further fine-tuned for the task of interest, show a large improvement over state-of-the-art.", "questions": "1. Why do the evaluation for code similarity detection use cosine similarity (l. 321) when the objective (l. 212) uses the l2 distance?\n2. What is the underlying metric for the Pass@k in the decompilation evaluation? Exact match, or some more lenient equivalent? It seems wrong to use exact match when, for instance, variable names would be arbitrary.\n3. In Table 4, the second row is exactly the \"Nova-1B\" row of Table 2, but I was under the impression that \"Nova-1B\" was more than just \"DeepSeekCoder + Nova's attention\", in particular the additional training data, and CL objective. Are the numbers off, or the caption, or did I miss something?\n4. When creating the assembly datasets (Appendix A.1), why go all the way to compiling executables, then using `objdump` for disassembling, with the associated possibilities of failure, rather than dump the assembly in the first place with `gcc -S`?\n5. Do you have preliminary results, citations, or intuition behind the \"normalizing\" step of the assembly language performed in Fig. 6, in particular the addition of spaces? Is that necessary?\n\nMinor points:\n1. In the numerator on l. 265, is $f_j^q$ supposed to be $f^q$? or $f_j^p$ for which the substitution wouldn't apply?\n2. l. 300, how many samples do the GPT models perform, then, to be able to compute the Pass@10 in Table 2?\n\nEdit after discussion and update\n--------------------------------------------\nThe overall clarity has improved, and additional information was provided.\nMost questions have been answered, so I'm raising my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality\n--------------\n1. While hierarchical attention mechanisms are not new, the design of this one is innovative in that: it takes into account the specific format and constraints of assembly instructions, and it accommodates for using regular tokens in the same sequence (e.g., natural text instructions).\n2. The contrastive objective losses, as well, encode a priori knowledge of the underlying data: compilation stages preserve semantics, and optimization stages are sequential.\n\nQuality\n----------\nThe different contributions are overall sensible, and contribute to the performance of the model. Experiments are adequately designed, and support the conclusions of the paper. The additional experiments help understand the role of the different contributions, in particular their effect on how embeddings get clustered and the effect it can have on the final model's performance.\n\nClarity\n---------\n1. The paper includes most of the relevant information, either in the main text or appendix. Relevant literature is mentioned and cited.\n2. Figures and examples make it much easier to understand the logic, especially Fig. 3.\n\nSignificance\n-----------------\n1. This work shows a significant improvement on benchmarks, sustained across model sizes, and adaptable to other models. This is an advancement on an important, developing field of machine learning applications.\n2. Given that these improvements do not require any in-depth change (e.g., to the vocabulary) and are compatible with already pre-trained model make it easier to experiment with in different settings.", "weaknesses": "Quality\n----------\n1. One of the 3 motivating cases in the introduction, malware detection, is not evaluated or considered at all in the rest of the paper. I understand the scope of the paper needs to end somewhere, but it would have strengthened the paper to include experiments on such a dataset.\n2. Details are missing in how the authors are certain that test data sets (both for decompilation and for similarity detection) do not overlap with any of the training data, including the pre-training data of DeepSeek-Coder, even inadvertently.\n3. An additional ablation on $\\textrm{Nova}_{-CL}$ would have helped see if there are any non-linear interactions between HA and CL.\n\nClarity\n---------\nThe overall organization of the paper could be improved. Many times, a concept, method, or setting is used in context before being formally explained. For instance:\n1. If the \"Related Work\" section is positioned earlier, it would help introduce the baseline models (DeepSeekCoder, LLM4Decompile) that are used in the previous \"Results\" section, as well as attention mechanisms, including LongCoder's, also used earlier.\n2. When describing the new datasets, it should be clear much earlier that \"source code\" really means \"C code\" (in the caption of Table 1, for instance), \"assembly\" is X86 assembly (or maybe X86-64? that's not so clear), that only `gcc` is considered as a compiler, and whether each \"program\" actually means a full executable program, or if it includes functions as well.\n3. Similarly, the contrastive losses mention \"the embedding\" of a function, which is quite ambiguous in transformers, especially if the model family (encoder-decoder?) is not mentioned.\n4. There is also a lot of ambiguity in notation, or the semantics of different objects. For instance:\n    * Do Table 1, and Appendix A.2, refer to the original \"AnghaBench\" and \"The-Stack\" datasets, or the new datasets constructed by the authors in Section 2.1? Maybe it would be better to name the new ones.\n    * In Functionality CL, l. 208 says it \"optimizes Nova with the constraint\", but a constraint is not a loss or objective. l. 215, \"constraints can be trained\" do not really make sense. It's also not obvious how the loss defined at l. 220 actually implements (a relaxation of) these constraints. It's also not explained if the sum over $f_i \\in F$ is actually done over all the million embeddings in the corpus, or how it's implemented in practice.\n    * K is introduced in Section 2.5, and then in 3.3., but we don't know what kinds of values will be used in practice. Also, Table 2 uses \"Pass@K\", but that's not the same K.\n    * In captions of Fig. 4 (b) and (d), the tables are more \"designs\" than \"implementations\"\n    * In Fig. 4 (b), the 1-4 indices are unfortunate as, for instance, $O0_3$ reads a lot like `-oO3`\n    * The equations at l. 220 and l. 266 have a really similar form, but the use of indices $i$ and $j$ is swapped between the two, making it a bit harder\n\nSignificance\n-----------------\nThe results are somewhat limited by the use of a single assembly language, and a single compiler, but this is acknowledged and does not seem like a fundamental limitation.\n\nMinor points\n-----------------\nl. 461: \"cauclated\" -> \"calculated\"?\n\nIn the bibliography:\n- Vaswani et al. is actually from 2017, not 2023 (though the arXiv version has had an inconsequential update in 2023), and a venue should be indicated (I'd suggest NeurIPS rather than arXiv)\n- Other articles are missing a venue or source\n- Several articles have incorrect capitalization in the title due to the lack of curly braces, e.g., use `{CodeT5}` to avoid it being rendered as \"Codet5\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mpa5psXQql", "reviewer": "Reviewer_hTSw", "summary": "This paper presents Nova, a generative language model specifically crafted for understanding and generating assembly code. Nova integrates hierarchical attention mechanisms with contrastive learning to effectively capture the semantics of code. The hierarchical attention mechanism focuses on intra-instruction, preceding-instruction, and inter-instruction relations, while contrastive learning ensures that functionally equivalent code, even with different optimizations, is similarly represented. The model is evaluated on two key tasks: decompilation (recovering high-level source code from assembly) and binary code similarity detection (BCSD) (measuring the similarity between binary code functions). Nova shows superior performance in both tasks, excelling in decompilation by accurately generating source code from optimized assembly, and achieving high recall in BCSD by effectively identifying similar code across different optimization levels.", "questions": "1. The paper argues that preceding-instruction attention helps avoid reuse of the same register (e.g., \"eax\") immediately after it is used in the previous instruction. However, this motivation is questionable because it does not explain how further subsequent instructions are prevented from reusing the same register. A more straightforward solution could be achieved with inter-instruction attention, as it can attend to all previous instructions, which raises the concern of functional overlap between preceding-instruction attention and inter-instruction attention, thus potentially making the preceding-instruction attention redundant.\n2. While Nova-1B and Nova-6B are much larger than CodeART, their performance gains in BCSD are limited. For example, in the k=100 case, CodeART sees a 17% improvement over JTrans with attention regularization, but Nova's improvement is only marginal, from 0.76 to 0.78 (as shown in Table 12). This suggests that adding hierarchical attention and other inductive biases provides limited benefits when scaling the model, and Tables 11-14 show that removing hierarchical attention does not lead to significant performance drops, questioning its overall necessity. And also in Table 5, the improvement brought by contrastive learning is much higher than the Hierarchical Attention.\n3. In the paper's analysis of attention distribution (Figure 10), the standard attention frequently converges on the first token, a phenomenon known as attention sink [1]. This behavior is also evident in the analysis of hierarchical attention (Figure 10(c, d)), where each token strongly attends to the first token within its attention mask, specifically the [INST-(x-1)] token, which represents the summary of the previous instruction. But it is not common when human try to interpret the functionality of each individual instruction. Furthermore, the justification for the Hierarchical Attention Mechanism —which selectively uses specific attention heads to represent the best attention maps—is somewhat ad hoc and lacks a clearer rationale.    \n4. The Hierarchical Attention Mechanism introduced in this paper represents a strong inductive bias; however, the underlying insights behind this inductive bias are not clearly explained. Additionally, the mechanism bears a striking resemblance to the Attention Regularization used in CodeART, with the primary difference being the absence of Preceding-Instruction Attention in CodeART. The effectiveness of this additional attention component has also been called into question earlier in the reivew, casting some doubt on its true contribution to the overall performance.\n5. While the use of contrastive learning aligns well with the BCSD task—improving performance by ensuring that functionally similar binaries, even across different optimizations, are represented similarly—it's less clear how this objective enhances the model's ability in decompliation. The training goal focuses on increasing the similarity of tokens from the same function but compiled with different optimization settings. However, this doesn't seem directly aligned with the ultimate goal of recovering executable source code, which requires more precise structural and semantic understanding beyond just token similarity across optimization levels. It would be greatly appreciated if the authors could provide some intuition as to why this approach can lead to improvements in decompliation.\n6. The authors introduced a novel optimization contrastive learning approach for the BCSD task, which had not been previously applied in the previous works, which commonly use the InfoNCE loss (line 220) or the triplet loss. As it is not discussed with deeper detail in the paper, it raises the question of whether these gains are substantial enough to justify the added complexity and whether this approach could be effectively generalized to improve other models in BCSD tasks.\n\n[1]: Efficient Streaming Language Models with Attention Sinks", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-structured and easy to follow. Concepts such as hierarchical attention and contrastive learning are clearly explained.\n2. The paper proposes a new method for encoding assembly code by using a Hierarchical Attention Mechanism to effectively capture the semantics of assembly instructions, while employing Contrastive Learning to ensure that functionally equivalent assembly code, even at different optimization levels, is represented similarly. This novel combination allows the model to robustly understand and learn from diverse assembly code structures.\n3. The paper conducts a broad range of experiments across multiple tasks and datasets, providing comprehensive evidence of the model’s effectiveness.\n4. Despite its specialized focus on assembly code, Nova's hierarchical attention is compatible with standard self-attention mechanisms, allowing it to seamlessly integrate and benefit from advancements in base models and code generation models.", "weaknesses": "1. Unclear motivation for introducing several inductive bias by Hierarchical Attention Mechanism. While the added attention mask inductive bias shows promising results in the BCD task, its impact in the BCSD task is minimal. This discrepancy raises questions about why the inductive bias performs well in one task but fails to offer significant improvements in the other.\n2. Lack of Design Discussion. The paper lacks sufficient discussion on key design components like Preceding-Instruction Attention and Optimization Contrastive Learning (CL). Without Preceding-Instruction Attention, the attention design is quite similar to CodeART, raising questions about the novelty and contribution of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldPyLwybT0", "reviewer": "Reviewer_n2JL", "summary": "This paper presents Nova, a generative language model specifically designed for assembly code, addressing unique challenges posed by the low information density and diversity in assembly syntax due to compiler optimizations. Nova introduces a hierarchical attention mechanism and employs contrastive learning to improve the model's understanding of assembly semantics across diverse optimization levels. Trained on a large assembly corpus, Nova outperforms existing techniques in tasks like binary code decompilation and binary code similarity detection, showing  improvements in Pass@1 and Recall@1 rates over state-of-the-art models.", "questions": "For binary similarity detection, compilers may inline functions or eliminate them altogether. How does your approach handle such scenarios?\n\nIf additional information (e.g., execution traces) were provided to GPT, or if iterative interaction with GPT were allowed, could the proposed approach still outperform a GPT-based model?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear Writing and Novel Application: The paper is well-written and easy to follow. The idea of applying hierarchical attention to assembly code is interesting and novel. While hierarchical attention is commonly used in NLP tasks, applying this mechanism to assembly code is, to the best of my knowledge, unprecedented.\n\n2. Promising Results: The evaluation results are promising. Nova demonstrates substantial improvements in both decompilation accuracy and similarity detection compared to existing models, validating its approach with strong experimental evidence.", "weaknesses": "Generalizability: The model is trained exclusively on x86 assembly code, which may limit its generalizability to other assembly languages, such as ARM or MIPS.\n\nRealism of Evaluation Settings:\n\n(1) The decompilation prompt requires optimization level information, but it is unclear if this information is accessible in stripped binaries.\n\n(2) For baseline models like GPT, fine-tuning with additional data isn’t necessary, raising questions about the fairness of the comparison. If GPT were given a few-shot learning setup or fine-tuned using OpenAI’s API, could it still be outperformed by the proposed approach?\n\n\nRelated Work: The paper omits discussion of several relevant works, which could provide a broader context for its contributions.\n\n[1] Debin: Predicting Debug Information in Stripped Binaries. CCS 2018\n\n[2] {DIRE}: A Neural Approach to Decompiled Identifier Renaming. ASE 2019\n\n[3] Learning to Reverse DNNs from AI Programs Automatically. IJCAI 2022\n\n[4] Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization. S&P 2019\n\n[5] Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection. CCS 2017.\n\n[6] ecompiling x86 Deep Neural Network Executables. Security 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLbFuEOuQl", "reviewer": "Reviewer_v7d8", "summary": "This paper introduces a generative model, Nova, tailored for assembly code tasks. Nova employs a hierarchical attention mechanism and is trained using contrastive learning objectives. This paper evaluates its effectiveness on two assembly code tasks.", "questions": "Please check my concerns in the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n+ The topic is interesting and important, addressing large language model (LLM) comprehension of assembly code.\n+ The paper is well-structured and easy to follow.", "weaknesses": "Weaknesses:\n- Comparison may be unfair due to different fine-tuning practices.\n- Evaluation of individual components is insufficient.\n- Generalization assessment is lacking.\n \n(1) Unfair Comparison: Nova is evaluated on two tasks, with fine-tuning applied specifically for each. However, the baseline models (such as Table 2) do not undergo the same fine-tuning for the tasks, leading to a potentially unfair comparison.\n \n(2) Component Evaluation: Nova’s hierarchical self-attention mechanism consists of three components, yet the paper lacks detailed performance assessments for each part. Despite a reasonable design, their individual impact remains unexamined.\n \n(3) Contrastive Learning Objectives: The contrastive learning objectives contain two distinct components. Further evidence is necessary to substantiate the utility of each objective. Additionally, the contrastive learning approach depends on the available optimization levels. Handling unseen optimization levels at inference should be discussed.\n \n(4) Normalization Process: In the data collection section, a normalization step is applied, but its relevance or benefit to Nova’s training is unclear.\n \n(5) Results across different optimization levels should be explored—e.g., training on O0, O1, O2 and testing on O3.\n \n(6) Random Sampling in BCSD Task: The BCSD task employs random sampling, yet statistical results are missing. Reporting such results would reduce the impact of randomness on performance claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nukDO1s3WE", "reviewer": "Reviewer_cjoF", "summary": "This paper presents Nova, a generative LLM for assembly code. To effectively learn from assembly with low information density, it uses a novel hierarchical attention mechanism which combines intra-instruction attention, preceding-instruction attention and inter-instruction attention. It further utilizes contrastive learning to better learn the semantics of assembly code from different optimization levels. The authors demonstrate the superiority of Nova over the baselines on binary code decompilation and code similarity detection tasks.", "questions": "1. See weakness 1,2 \n2. Could you provide more details about how to construct $F$ in practice used in Functional CL?\n3. The authors state that hierarchical attention is only applied to half of the attention head. Since different attention heads can learn different features, I wonder if this setup is robust to the selection of the attention heads?\n4. Would the pre-trained models (Nova-1B, 6B) be public available?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. LLMs for binary code is an important topic to study\n2. This work proposes new methods to train Nova based on the properties of assembly code, which is clearly motivated.\n3. The proposed models show a clear improvement in binary code decompilation.", "weaknesses": "1. The comparison on code similarity detection may not be fair. For example, CodeArt uses 12 transformer blocks with 768 hidden dimensions, whose size is smaller than Nova-1B. The authors should compare Nova with the baseline under a similar size with the same pre-training data to demonstrate the superiority of Nova on code similarity detection. For the current result, we can find that compared with CodeArt, Nova actually does not show a significant improvement (e.g. both are 0.64 for Nova-1B under K=500). So it is in question whether Nova is indeed better for code similarity detection.\n2. The experiments for Comparison with Techniques Handling Long Input are confusing. Specifically, it has the following problems:\n\na) What is \"Nova’s Fine-Tuning\" in Table 3? It seems Nova does not have something special in terms of fine-tuning. Does it just mean fine-tuning with hierarchical attention or also with Nova's pretraining as suggested in Line 360? \n\nb) What is the average token length for downstream tasks before truncation? The authors want to claim Nova is better at solving long input challenges. But I see from the Appendix that Nova uses the input length as 1024 tokens during pre-training and 2048 for fine-tuning. It may be hard to claim this length to be \"long-context\". Considering that assembly code should be much longer than source code and Granite-3B-Code-128K can handle 128K input tokens at most, have you tested in the benchmarks where the input context is longer, e.g. 8k/32k/128k?\n\n3. The presentation of the paper can be improved. Specifically, a) Line 281 is unclear. The authors should clearly state that their pre-training contains two stages and the loss in Line 240 is used in the second stage. b) The ablation study should be separated into new subsections instead of mixing with Section 4.1 c) The equations are not numbered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)