Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning

## Abstract

Abstract Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively, and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova , a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 – 21.58% (absolute percentage point improvement) higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks.

## 1 Introduction

Binary code plays an irreplaceable role in the security domain, being the foundation of crucial tasks including vulnerability detection (Güler et al., ; Duan et al., ; Chen et al., ), malware detection (Spensky et al., ; Aonzo et al., ; Xu et al., ), binary recovery (Su et al., ; Zhang et al., ; Chen et al., ), and legacy software maintenance (Carbone et al., ; Carlini et al., ; Martin et al., ).
For example, when performing tasks such as identifying attacks and malware, security analysts often only have access to assembly, i.e., the human-readable representation of binary code, which is extremely difficult to understand (Su et al., ; Zhang et al., ; Chen et al., ).
Thus, combined with the increasing sophistication of cybercrime that poses significant threats worldwide (e.g., cybercrime is predicted to cost the world $10.5 trillion annually by 2025 (Sausalito, )), effective binary analysis techniques are in high demand.

Figure: Figure 1: Example that shows the semantics and diverse optimizations of assembly code.
Refer to caption: x1.png

Large language models pre-trained on source code have brought improvement in various software development domains (Chen et al., ; Liu et al., ; Chen et al., ; Le et al., ; Jiang et al., ; Xia et al., ). However, these LLMs are not designed for or trained with assembly corpus, not achieving their full potential on binary code analysis tasks such as binary code similarity (Wang et al., ; Xu et al., ), malware detection (Su et al., ), and binary code decompilation (Tan et al., ; Armengol-Estapé et al., ; Hosseini & Dolan-Gavitt, ).

Existing work applying LLMs on assembly code mainly piggybacks on encoder-style LLMs (Wang et al., ; Su et al., ; Xu et al., ), unable to benefit from the more extensive pre-training, updated architectures, scaling of state-of-the-art generative LLMs. Other work using generative LLMs for decompilation shows a low unit test passing rate of the decompiled programs (Tan et al., ; Armengol-Estapé et al., ).

The challenges of leveraging generative LLMs for assembly code are twofold. First, compared to source code, assembly code has a *lower information density*.
A short source-code sequence maps to an assembly-code sequence that is often several times longer. Thus, assembly semantics span across a *long sequence of tokens*.
Figure 1 (a) shows an example of a source code function that compares two integers, while Figure 1 (b) shows its corresponding assembly code optimized with O0 flag. In the O0-optimized assembly code, the five instructions from 10:   mov   -0x8(%rbp),%rax to 1c:   cmp   %eax,%edx perform the checking whether the value of x is smaller than the value of y (correspond to if (*(int*)x < *(int*)y) in the source code). A single assembly instruction alone represents little meaningful semantics in the source code. It is the combinations of *many* *instructions* and the *dependencies* between them represent the semantics. Such combinations of instructions are long, which is hard for LLMs to learn.

Second, assembly code is diverse due to compiler optimization. The assembly code of the same source code function looks dramatically different with different compiler optimization.
Figure 1 (c) shows the assembly of the same function compiled with O1 and O0 flags, which consists of a significantly different set of instructions. Such syntax diversity is hard for LLMs to learn, preventing LLMs from obtaining consistently good performances on differently optimized assembly code.

In this work, we develop Nova, a generative foundation LLM pre-trained for assembly code with two key novelties. First, to address the low-information-density and long-sequence challenge, we design a hierarchical self-attention, which contains three categories of attention at different levels of granularity: intra-instruction attention, preceding-instruction attention, and inter-instruction attention. The key insight is to build *attention summaries*, i.e., we create per-statement attention *labels*, which act as the summary of a statement. We then use preceding-instruction attention to capture semantics between a token and its preceding instruction label and use inter-instruction attention for long dependencies. Besides, we design *functionality contrastive learning* and *optimization contrastive learning* objectives to train Nova to learn the semantics behind the diverse syntax of assembly.

This work makes the following contributions:

- •
We propose a novel hierarchical attention mechanism that captures the assembly’s low-density semantics at three granularity levels.
- •
We design contrastive learning objectives to train LLMs to learn assembly with diverse optimizations and encode assembly more efficiently.
- •
We develop Nova, a generative foundation LLM with hierarchical attention and contrastive learning for assembly. Nova outperforms state-of-the-art on binary decompilation by up to 14.84 – 21.58% higher Pass@k and on binary similarity detection by up to 6.17% Recall@1.
- •
Availability: we release Nova models at [https://huggingface.co/lt-asset/nova-6.7b](https://huggingface.co/lt-asset/nova-6.7b) and [https://huggingface.co/lt-asset/nova-6.7b-bcr](https://huggingface.co/lt-asset/nova-6.7b-bcr)

## 2 Approach

Figure 2 presents the overall approach of Nova. We build Nova on top of foundation models for source code (Guo et al., ) to utilize their source code and natural language generation ability.
We first collect large assembly corpora (Section 2.1).
Section 2.2 describes Nova’s hierarchical attention design.
With the collected assembly corpora, we then pretrain Nova with language modeling and contrastive learning objectives (Section 2.3).
Then, we fine-tune Nova on two important downstream tasks, binary code decompilation, and binary code similarity detection (Sections 2.4 and 2.5), to prove Nova’s effectiveness and benefits to the binary research domain.

Figure: Figure 2: Overview of developing Nova

Table: Table 1: Statistics (number of source code and assembly functions) of the pre-training datasets.

| Datasets | Source | O0 | O1 | O2 | O3 | Total |
| --- | --- | --- | --- | --- | --- | --- |
| AnghaBench | 757.1K | 743.1K | 726.4K | 718.7K | 717.8K | 3.7M |
| The-Stack | 138.8K | 125.1K | 119.7K | 116.9K | 108.8K | 609.3K |

### 2.1 Data Collection

We build our assembly data sets on top of existing source code datasets: The-Stack (Li et al., ) and the AnghaBench (da Silva et al., ). We compile the source code into executables with different optimization levels (i.e., O0, O1, O2 and O3), strip the executables to remove debug information, and disassemble them into assembly code. We treat every assembly function as a separate data sample. The breakdown statistics are in Table 2.

We perform certain normalization on the assembly functions: (1) removing all the “%” and comments, (2) adding whitespace around “,”, “(”, “)”, (3) converting all the hexadecimal numbers to decimal numbers, and (4) replacing the address of each instruction with special labels (e.g., replacing “0” and “4” in Figure 1 (b) with “[INST-1]” and “[INST-2]”) placing at the end of each instruction. More details are in Appendix A.1.

### 2.2 Hierarchical Self-Attention

Nova uses hierarchical self-attention that is specially designed to learn the *low-information-density* semantics in the *long* sequence of assembly code. Specifically, Nova learns the assembly code in an
hierarchical way by providing a modified attention mask. Different from standard token-level attentions (Vaswani et al., ; Radford & Narasimhan, ; Radford et al., ; Brown et al., ),
our hierarchical self-attention contains three categories at different levels of granularity.

Figure: Figure 3: Design of Nova’s hierarchical attention for assembly code
Refer to caption: x3.png

(1) Intra-Instruction Attention:
Due to the low information density in assembly, intra-instruction attention is designed to capture the summary of every instruction, which is the standard causal attention but limited to tokens of each instruction (the yellow part in Figures 3) (a) and (b).
Tokens in different instructions have no attention weights. The “
” label at the end of the instruction has attention to all the tokens in the instruction and thus captures the semantics of the entire instruction (e.g., “[INST-1]” captures the semantics of “mov eax, $1”).

(2) Preceding-Instruction Attention:
In addition to the local semantics of each instruction, the use of assembly instructions (such as the choice of registers) depends on the context.
For example, after the first instruction “mov eax, $1”, the second instruction should not reuse “eax” to store another value “$2” immediately. To capture such context, the preceding-instruction attention enables each token in an instruction to have attention to the “
” label of the preceding instruction (the light green part in Figures 3 (a) and (b)).

(3) Inter-Instruction Attention:
To understand function semantics (i.e., functionality), which lies in the dependencies across different instructions, the inter-instruction attention is designed to let the “
” label of each instruction have attention to all the labels of previous instructions. For example, “[INST-4]” has attention to “[INST-1]”, “[INST-2]”, and “[INST-3]” (the dark green part in Figures 3 (a) and (b)). The inter-instruction attention is only enabled for “
” labels, as they represent the semantics of each instruction.

To sum up, the hierarchical self-attention breaks the semantics of assembly code into three parts. The intra-instruction attention captures the instruction summary, and the preceding-instruction attention captures the context with the preceding instruction. The inter-instruction attention learns the long dependencies across instructions on top of the “
” labels that contain the instruction summary. Figure 3 (c) also shows how Nova’s hierarchical attention for assembly code is compatible with the standard self-attention for text and source code. As existing LLMs have shown good performance on text and source code using the standard self-attention, we keep the standard causal attention mask within and between any chunks of text or source code in the input (the light grey part shown in Figure 3 (c)). The attention from text or source code to assembly code (and vice versa) is only allowed for the “
” tokens as they are designed to contain the assembly instruction summaries.

### 2.3 Contrastive Learning

The syntax gap between assembly code and source code, and syntax diversity between differently-optimized assembly code make LLMs struggle to distinguish the semantics behind the syntax. Nova adopts contrastive learning technique (Gao et al., ) during pre-training to train LLMs to encode assembly code meaningfully w.r.t semantics.
The standard pre-training objective is language modeling by minimizing the negative likelihood of code in the pre-training corpus (Radford & Narasimhan, ), notated as $L_{lm}$. In addition, Nova is pre-trained with two new objectives, $L_{fcl}$ for functionality contrastive learning and $L_{ocl}$ for optimization contrastive learning.

Figure: Figure 4: Design of functionality and optimization contrastive learning (CL). “asm” denotes assembly.
Refer to caption: x4.png

Functionality CL:
Functionality CL trains Nova to focus more on the functionalities of assembly code rather than the syntax. Code with the same functionality (assemblies from the same source code), should be encoded closer in the latent space. For instance, in Figure 4 (a), embeddings of source and assembly code of function “cmp” are closer to each other, and the same for function “sort”.

Let $e^{s}_{f}$ be the embedding of function $f$ in $s$ form ($s=-1$ for source code, and $s\in[0,1,2,3]$ for O0 to O3 optimized assembly). For simplicity, let $S=[-1,0,1,2,3]$ be the domain of $s$. We use the average of all the “
” tokens’ embedding as the embedding of the whole assembly function, as each “
” token is supposed to capture the semantics of that instruction by the design of hierarchical self-attention. Functionality CL optimizes Nova with the constraint:

$$ $\forall f_{i}\in F,\;\mathop{max}\limits_{s,t\in S}(\mathop{d}(e^{s}_{f_{i}},e ^{t}_{f_{i}}))<\mathop{min}\limits_{\begin{subarray}{c}s,t\in S\\ f_{j}\neq f_{i}\in F\end{subarray}}(\mathop{d}(e^{s}_{f_{i}},e^{t}_{f_{j}}))$ $$

, where $\mathop{d}$ calculates the $l_{2}$ distance between two embeddings and $F$ is the full set of functions in the training corpus.

Such constraints can be trained by optimizing the embeddings of a batch of functions, each function in two different forms. For the example in Figure 4 (b), there are two forms (source code and O0 assembly) of four functions. Once Nova encodes the batch of source code and assembly functions, we calculate the distance matrix $\{D_{ij}\}_{f_{i},f_{j}\in F}=\{d(e^{s}_{f_{i}},e^{t}_{f_{j}})\}$, and minimize the loss:

$$ $L_{fcl}=-\mathop{log}\sum_{s,t\in S}\sum_{f_{i}\in F}\left(1-\frac{\mathop{exp }\left(d(e^{s}_{f_{i}},e^{t}_{f_{i}})\right)}{\sum_{f_{j}\in F}\mathop{exp} \left(d(e^{s}_{f_{i}},e^{t}_{f_{j}})\right)}\right)$ $$

This objective minimizes the distance between embeddings for the same function, i.e., the diagonal in the distance matrix.

Optimization CL: LLMs can be confused if being asked to directly connect a source code function to its O3-optimized assembly, due to their dramatically different syntax. Such a huge gap can be filled by learning how the source code is transformed to O0, O1, O2 and eventually to O3 assembly, as the optimization levels are *ordered*.

Higher-level optimization applies a super-set of optimization rules compared to lower-level optimization. Nova learns such order with the optimization CL objective, encoding differently-optimized assembly code orderly. Optimization CL optimizes Nova with the constraint:
the more optimizations applied, the larger the difference between embeddings of optimized and unoptimized code. For instance, Figure 4 (c) and (d) illustrate that for the same function “cmp”, the distance between source code and assembly increases when the optimization level increases. Formally, optimization CL minimizes the following loss:

$$ $L_{ocl}=\sum\limits_{f\in F}\sum\limits_{s<t_{1}<t_{2}\in S}\mathop{max}\left( 0,d(e^{s}_{f},e^{t_{1}}_{f})-d(e^{s}_{f},e^{t_{2}}_{f})\right)$ $$

Overall, the final training loss combines the three: $L=L_{lm}+\lambda(L_{fcl}+L_{ocl})$, where $\lambda$ is set to 0.1 to balance the losses in this work.

### 2.4 Task 1: Binary Code Decompilation

Binary code decompilation (BCD) helps developers to understand binary code by recovering binary code into more readable high-level source code (e.g., C programs) (Fu et al., ; Liang et al., ; Armengol-Estapé et al., ; Tan et al., ). The input to the model for BCD is formatted as an instruction prompt (notated by $\mathbf{p}$): # This is the assembly code with {opt} optimization: {$\mathbf{asm}$}, where “opt” is the optimization-level applied to the assembly and “$\mathbf{asm}$” is the assembly code to decompile. Nova is fine-tuned to generate the expected source code function $\mathbf{src}$ following the instruction prompt. The fine-tuning objective is minimizing the loss: $L_{bcd}=-\mathop{log}\mathop{P}(\mathbf{src}|\mathbf{p})$.

### 2.5 Task 2: Binary Code Similarity Detection

Binary code similarity detection (BCSD) aims to measure the similarity between two binary code snippets (Wang et al., ; Su et al., ), which is the foundation of various applications such as plagiarism detection (Luo et al., ; Sæbjørnsen et al., ) and vulnerability detection (David & Yahav, ; David et al., ; ; ).

A widely used setting is taking a query assembly of the function $f^{q}$ that is compiled with one optimization level (denoted by $s$), and a pool of candidate assembly of $K$ functions (notated by $f^{p}_{i},$ $1\leq i\leq K$) compiled with a different optimization level (denoted by $t\neq s$). There exists a unique candidate assembly coming from the same source code as the query ($\exists!1\leq i\leq K,$ $f^{p}_{i}=f^{q}$, called the positive candidate). Nova is fine-tuned to encode these binaries, so that the positive candidate has the highest similarity with the query assembly among the pool. The learning objective is as follows:

$$ $L_{BCSD}=-\mathop{log}\sum\limits_{1\leq j\leq K}^{f^{q}:=f^{p}_{j}}\left(1- \frac{\mathop{exp}\left(\mathop{d}(e^{s}_{f^{q}},e^{t}_{f^{q}_{j}})\right)}{ \sum_{1\leq i\leq K}\mathop{exp}\left(\mathop{d}(e^{s}_{f^{q}},e^{t}_{f^{p}_{i }})\right)}\right)$ $$

We follow previous work (Su et al., ) to let $s$ be O0-assembly and $t$ be O3-assembly, which is the hardest setting.

## 3 Experimental Setup

This section describes the setup of pre-training and fine-tuning of Nova, as well as the existing baselines we compare Nova with, and the evaluation metrics we used in the two downstream tasks.
Appendix A.2 contains additional details such as training hyper-parameters.

### 3.1 Pre-Training

We use the data collected from AnghaBench and The-Stack for pre-training. We pre-train Nova starting from DeepSeek-Coder (Guo et al., ), and the hierarchical attention is applied on half of the attention heads to balance between its effectiveness and the existing knowledge in the standard attention layers (Justified in Appendix A.3). Nova is pre-trained with language modeling for one epoch, followed by contrastive learning objectives for another epoch.

### 3.2 Fine-Tuning for Binary Code Decompilation

Training Data: We sample (due to computation resource limitation) 2.16M assembly-to-source-code pairs (0.338B tokens) from the pre-training corpus to build the BCD fine-tuning data.

Test Data: We use HumanEval-Decompile (Tan et al., ) as the test benchmark, which was not used in training.
HumanEval-Decompile is derived from the C language adaptation of the HumanEval (Chen et al., ) benchmark and contains 164 C functions, each compiled with O0 – O3 optimization flags and disassembled into X86-64 assembly.

Baselines: Nova is compared with
existing SOTA LLM4Decompile (Tan et al., ), Meta LLMCompiler (Cummins et al., ), and other general code LLMs (CodeLlama (Rozière et al., ), StarCoder2 (Lozhkov et al., ), DeepSeekCoder (Guo et al., ), Qwen2.5-Coder (Hui et al., ), Granite Mishra et al. (), LongCoder (Guo et al., ), GPT-3.5-Turbo, and GPT-4o). LLM4Decompile trains DeepSeekCoder using the same AnghaBench corpus for binary decompilation. Meta LLMCompiler trains CodeLlama models using LLVM IRs, X86, and ARM assembly code to optimize and translate binary code.

Evaluation: GPT-3.5-Turbo and GPT-4o are prompted with three-shot examples, while all the other LLMs and Nova sample 20 decompilations per assembly code, using the temperature of 0.2 and top_p of 0.95 (Chen et al., ). The generated decompilations are executed against the test cases. Finally, Pass@1 and Pass@10 (Chen et al., ) are reported.

### 3.3 Fine-Tuning for Binary Code Similarity Detection

Training Data: To compare Nova with existing works on BCSD fairly (Wang et al., ; Su et al., ), we use the same dataset, BinaryCorp-3M (Wang et al., ), as the fine-tuning data for BCSD, which contains the O0 and O3 assembly of 224,606 functions.

Test Data: Following existing work (Su et al., ; Xu et al., ), we use real-world benchmarks, Binutils, Curl, ImageMagick, SQLite, OpenSSL, and Putty, as the test benchmarks, which are nonexistent in the training data.

Baselines: Nova is compared with jTrans (Wang et al., ), DiEmph (Xu et al., ) and CodeArt (Su et al., ). jTrans is a Transformer (Vaswani et al., ) encoder trained on binaries with masked token prediction and jump target prediction tasks. DiEmph uses an instruction deemphasis technique to prevent the model from learning instruction
distribution biases introduced by compilers. CodeArt proposes a regularized attention mask for encoder models to capture instructional semantics and data dependencies.

Evaluation: We randomly sample $K$ source code functions from each project, compile them into binaries with O0 and O3 optimization flags, and disassemble them into X86-64 assemblies. BCSD techniques encode these assemblies into embeddings (Nova uses the average last-layer hidden states of all the “
” tokens in an assembly as its embedding). Then each O0 assembly is used as the query to calculate their similarity with the $K$ O3 candidate assemblies (using cosine similarity). Metric Recall@1 is the ratio of queries for which the candidate from the same source code has the highest similarity among all the candidates.

## 4 Results

### 4.1 Binary Code Decompilation

Comparison with SOTA Techniques:
Table 2 shows the Pass@1 and Pass@10 of the decompiled code from assemblies on HumanEval-Decompile. The results are grouped by optimization level (i.e., the benchmark contains 164 assemblies of each optimization level), and the average is also reported.

**Table 2: Nova’s Pass@K and comparison with existing techniques on HumanEval-Decompile.**
| Techniques | Pass@1 | Pass@10 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O0 | O1 | O2 | O3 | Avg. | O0 | O1 | O2 | O3 | Avg. |  |
| CodeLlama-7B | 6.95 | 3.81 | 4.54 | 3.78 | 4.77 | 8.53 | 5.97 | 7.34 | 5.17 | 6.75 |
| StarCoder2-7B | 6.31 | 4.33 | 5.64 | 5.95 | 5.56 | 8.77 | 5.18 | 6.09 | 7.17 | 6.80 |
| DeepSeekCoder-7B | 9.63 | 7.56 | 7.41 | 6.68 | 7.82 | 13.60 | 11.38 | 11.52 | 9.44 | 11.49 |
| Qwen-2.5-Coder-7B | 4.76 | 5.79 | 5.58 | 5.27 | 5.35 | 6.34 | 7.69 | 6.56 | 5.79 | 6.60 |
| LLMCompiler-7B | 5.95 | 5.85 | 5.55 | 5.82 | 5.79 | 7.01 | 7.31 | 7.47 | 7.01 | 7.20 |
| GPT-3.5-Turbo | 7.41 | 6.13 | 4.33 | 3.90 | 5.44 | 9.56 | 8.38 | 6.23 | 5.12 | 7.32 |
| GPT-4o | 21.34 | 18.29 | 14.48 | 13.05 | 16.79 | 29.94 | 26.74 | 21.42 | 19.88 | 24.50 |
| LLM4Decompile-1B | 15.30 | 8.26 | 9.36 | 8.38 | 10.33 | 21.79 | 15.23 | 16.17 | 13.70 | 16.72 |
| Nova-1B | 37.53 | 21.71 | 22.68 | 18.75 | 25.17 | 49.38 | 34.84 | 36.95 | 32.03 | 38.30 |
| LLM4Decompile-6B | 29.97 | 19.05 | 20.46 | 18.32 | 21.95 | 40.40 | 27.75 | 28.85 | 28.51 | 31.38 |
| Nova-6B | 48.78 | 30.58 | 30.85 | 27.23 | 34.36 | 57.47 | 47.45 | 43.03 | 39.68 | 46.91 |

*Overall, Nova’s Pass@1 and Pass@10 are higher than all SOTA binary decompilation techniques and general LLMs with even smaller model sizes*.
Specifically, for each optimization level, Nova consistently decompiles more assemblies into source code correctly than the rest of the compared techniques. Note that Meta LLMCompiler is mainly designed for LLVM IR optimization, and thus is still incapable of assembly code decompilation.

With the same model size, Nova-1B outperforms LLM4Decompile-1B with a 14.84% higher averaged Pass@1, and a 21.58% higher Pass@10. Nova-6B outperforms LLM4Decompile-6B with a 12.41% higher averaged Pass@1, and a 15.53% higher Pass@10. When compared with GPT-4o, an order of magnitude larger model, Nova-1B produces an 8.38% higher Pass@1 and 13.80% higher Pass@10.
Examples of Nova’s correct decompilation are provided in Appendix A.4.

Comparison with Techniques Handling Long Input:
Nova’s hierarchical attention design targets to address the low information density and long input challenge of assembly code. There are other techniques that handles long input challenges in text and source code, with Granite-3B-Code-Base-128K and LongCoder being the most related ones.
Granite trains LLM on repository-level long inputs, which is an orthogonal approach with Nova’s approach (hierarchical attention and contrastive learning). We train Granite-3B-Code-128K with Nova’s approach, and Table 3 shows that Nova’s approach brings improvement to Granite over standard fine-tuning even if it has already been trained with long code data.

**Table 3: Nova’s approach brings improvement to LLM that has been trained with long input data.**
| Techniques | Pass@1 | Pass@10 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O0 | O1 | O2 | O3 | Avg. | O0 | O1 | O2 | O3 | Avg. |  |
| Granite (3B-Code-128K) | 5.91 | 3.78 | 5.09 | 5.52 | 5.08 | 8.19 | 5.16 | 6.56 | 7.15 | 6.76 |
| Granite + Standard Fine-Tuning | 20.88 | 13.54 | 11.37 | 10.09 | 13.97 | 30.05 | 19.77 | 18.31 | 15.77 | 20.98 |
| Granite + Nova’s Fine-Tuning | 31.04 | 14.57 | 14.70 | 13.66 | 18.49 | 39.57 | 21.23 | 21.77 | 19.82 | 25.60 |

**Table 4: Nova’s hierarchical attention is more effective on assembly code.**
| Techniques | Pass@1 | Pass@10 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O0 | O1 | O2 | O3 | Avg. | O0 | O1 | O2 | O3 | Avg. |  |
| DeepSeekCoder + LongCoder’s Attention | 34.59 | 19.07 | 19.72 | 17.34 | 22.68 | 42.19 | 32.37 | 32.86 | 29.04 | 34.12 |
| DeepSeekCoder + Nova’s Attention | 37.53 | 21.71 | 22.68 | 18.75 | 25.17 | 49.38 | 34.84 | 36.95 | 32.03 | 38.30 |

LongCoder combines window attention and global attention to learn long code input. We compare LongCoder’s attention design with Nova’s hierarchical attention design using DeepSeekCoder-1.3B as the backbone. Table 4 shows that Nova’s hierarchical attention is more effective in learning assembly code. Nova’s attention design considers the instruction-level local semantics and dependencies between different instructions, which fits better than fix-sized window attention to assembly code.

Ablation Study:
We conduct an ablation study by comparing Nova-1B with the following models to show the effectiveness of contrastive learning objectives and hierarchical attention design:

- •
Nova-CL-HA: Removing contrastive learning and hierarchical self-attention. This is simply training DeepSeekCoder-1.3B on the assembly corpus using language modeling. This can be viewed as our reproduction (retrain) of LLM4Decompile-1B using the same amount of data.
- •
Nova-HA: Removing the hierarchical self-attention, training DeepSeekCoder-1.3B on the assembly corpus using both the language modeling and contrastive learning objectives.

**Table 5: Ablation study of Nova-1B on HumanEval-Decompile.**
| Techniques | Pass@1 | Pass@10 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O0 | O1 | O2 | O3 | Avg. | O0 | O1 | O2 | O3 | Avg. |  |
| LLM4Decompile-1B | 15.30 | 8.26 | 9.36 | 8.38 | 10.33 | 21.79 | 15.23 | 16.17 | 13.70 | 16.72 |
| Nova-CL-HA | 20.73 | 16.16 | 15.03 | 11.19 | 15.78 | 33.55 | 28.12 | 26.96 | 21.01 | 27.41 |
| Nova-HA | 30.58 | 19.88 | 20.58 | 16.40 | 21.86 | 44.75 | 33.13 | 33.31 | 29.82 | 35.25 |
| Nova | 37.53 | 21.71 | 22.68 | 18.75 | 25.17 | 49.38 | 34.84 | 36.95 | 32.03 | 38.30 |

Table 5 shows the results of the ablation study. Nova-CL-HA produces an average Pass@1 of 15.78% and Pass@10 of 27.41%. With additional contrastive learning objectives, Nova-HA improves the Pass@1 on all optimization levels over Nova-CL-HA, showing a higher averaged Pass@1 and Pass@10. Further applying the hierarchical self-attention on Nova-HA boosts the overall Pass@1 from 21.86% to 25.17%, and Pass@10 from 35.25% to 38.30%.

**Table 6: Recall@1 on BCSD with $K=50$**
| Table 6: Recall@1 on BCSD with $K=50$<br>Benchmarks<br>jTrans<br>DiEmph<br>CodeArt<br>Nova-1B<br>Nova-6B<br>Binutils<br>0.68<br>0.80<br>0.84<br>0.87<br>0.89<br>Curl<br>0.72<br>0.84<br>0.86<br>0.89<br>0.94<br>ImageMagick<br>0.53<br>0.71<br>0.78<br>0.86<br>0.90<br>SQLite<br>0.73<br>0.79<br>0.78<br>0.77<br>0.78<br>OpenSSL<br>0.70<br>0.83<br>0.88<br>0.90<br>0.92<br>Putty<br>0.63<br>0.72<br>0.69<br>0.72<br>0.71<br>Avg.<br>0.67<br>0.78<br>0.81<br>0.84<br>0.86 | Table 7: Recall@1 on BCSD with $K=100$<br>Benchmarks<br>jTrans<br>DiEmph<br>CodeArt<br>Nova-1B<br>Nova-6B<br>Binutils<br>0.60<br>0.63<br>0.81<br>0.79<br>0.79<br>Curl<br>0.63<br>0.80<br>0.82<br>0.86<br>0.88<br>ImageMagick<br>0.54<br>0.71<br>0.76<br>0.79<br>0.81<br>SQLite<br>0.62<br>0.72<br>0.74<br>0.73<br>0.72<br>OpenSSL<br>0.60<br>0.80<br>0.87<br>0.88<br>0.90<br>Putty<br>0.58<br>0.64<br>0.64<br>0.65<br>0.64<br>Avg.<br>0.60<br>0.72<br>0.77<br>0.78<br>0.79 |
| --- | --- |
| Table 8: Recall@1 on BCSD with $K=200$<br>Benchmarks<br>jTrans<br>DiEmph<br>CodeArt<br>Nova-1B<br>Nova-6B<br>Binutils<br>0.51<br>0.64<br>0.74<br>0.73<br>0.73<br>Curl<br>0.57<br>0.77<br>0.78<br>0.83<br>0.84<br>ImageMagick<br>0.39<br>0.51<br>0.67<br>0.73<br>0.75<br>SQLite<br>0.56<br>0.65<br>0.68<br>0.68<br>0.69<br>OpenSSL<br>0.54<br>0.71<br>0.82<br>0.84<br>0.88<br>Putty<br>0.49<br>0.58<br>0.55<br>0.55<br>0.58<br>Avg.<br>0.51<br>0.64<br>0.71<br>0.73<br>0.75 | Table 9: Recall@1 on BCSD with $K=500$<br>Benchmarks<br>jTrans<br>DiEmph<br>CodeArt<br>Nova-1B<br>Nova-6B<br>Binutils<br>0.40<br>0.57<br>0.70<br>0.65<br>0.67<br>Curl<br>0.43<br>0.62<br>0.69<br>0.73<br>0.76<br>ImageMagick<br>0.25<br>0.42<br>0.58<br>0.61<br>0.65<br>SQLite<br>0.43<br>0.59<br>0.62<br>0.59<br>0.62<br>OpenSSL<br>0.43<br>0.61<br>0.76<br>0.78<br>0.82<br>Putty<br>0.38<br>0.50<br>0.49<br>0.47<br>0.51<br>Avg.<br>0.39<br>0.55<br>0.64<br>0.64<br>0.67 |

### 4.2 Binary Code Similarity Detection

Tables 9,  9,  9 and  9 show the Recall@1 of Nova and existing BCSD techniques with pool size $K$ of 50, 100, 200 and 500 on the six benchmarks.
Underline indicates the best in each benchmark, while wave denotes the tied best (we only mark Nova-1B for clearer illustration).

Overall, Tables 9,  9,  9 and 9 show that *on average, Nova-1B and Nova-6B achieve the highest Recall@1 (in bold) under all four settings of $K$*.
Nova-6B further outperforms Nova-1B and achieves the highest averaged Recall@1 under all four settings, ranking the ground-truth of 5%, 2%, 4%, and 3% more queries the most similar correspondingly compared to CodeArt.
Nova-1B consistently outperforms existing techniques with higher Recall@1 when $K$ is 50, 100, and 200, meaning it correctly ranks ground-truth of 3%, 1%, and 2% more queries as the most similar. Under the setting of $K=500$, Nova-1B ties with CodeArt with the same highest Recall@1. When looking into each individual benchmark, Nova-1B always wins on the most benchmarks under different settings of pool size $K$. For instance, Nova-1B wins on four benchmarks while DiEmph only wins on SQLite when $K=50$.
We also conduct an ablation study on BCSD in Appendix A.5.

### 4.3 Analytic Experiments: How are Nova’s embeddings better?

We use the widely-used t-SNE (van der Maaten & Hinton, ) to analyze and visualize high-dimensional embeddings. We randomly sample seven coding problems from HumenEval-Decompile (task_id 19, 32, 34, 63, 119, 128, 143), encode the O0 – O3 assemblies by Nova-CL-HA and Nova-1B. Figure 5 shows the embeddings that are visualized under the first two principal components. Each color represents one task, and O0 – O3 assemblies are marked by
$\bigcirc$
, $\bigtriangledown$, $\bigtriangleup$, and $\square$.

Compared with Nova-CL-HA (Figure 5 (a)), Nova-HA (Figure 5 (b)) including contrastive learning objectives in the pre-training, can separate the embeddings of assemblies with different functionalities better. Nova-HA clearly encode “Task 143” (orange points) away from the others. Yet, Nova’s (Figure 5 (c)) embeddings group the assemblies by functionalities more precisely than Nova-HA, suggesting that hierarchical attention enhances the training of contrastive learning objectives to learn more effective encoding.
Visualization using a different approach, PCA, is shown in Appendix A.6. Analytic experiments on Nova’s hierarchical attention is shown in Appendix A.7.

Figure: Figure 5: t-SNE analysis of embeddings cauclated by Nova-CL-HA, Nova-HA, and Nova.
Refer to caption: x5.png

## 5 Related Work

### 5.1 Binary Models

Machine learning models are widely used in binary program analysis tasks. However, these models are typically designed for specific tasks such as binary code similarity detection (Pei et al., ; Xu et al., ; Wang et al., ), variable name prediction (Chen et al., ; Xu et al., ; Zhang et al., ), binary code type inference (Pei et al., ), and are built from scratch.

Recent techniques have started to pre-train foundation LLMs for binaries. CodeArt (Su et al., ) pre-trains encoder-style LLMs with a regularized attention design to better encode assembly code semantics. SLaDe (Armengol-Estapé et al., ) trains BART (Lewis et al., ) models on assembly.
Meta LLMCompiler (Cummins et al., ) train CodeLlama models on LLVM IR to optimize binary code.
LLM4Decompile (Tan et al., ) trains DeepSeekCoder with assembly for binary code decompilation. However, CodeArt does not generalize to generation tasks due to its encoder architecture. LLMCompiler trained on LLVM IR cannot be effectively transferred to assembly code. SLaDe and LLM4Decompile are limited in performance due to a lack of special designs for assembly. In contrast, Nova proposes hierarchical attention and contrastive learning objectives, outperforming existing techniques on both understanding (binary code similarity detection) and generation (binary code decompilation) tasks.

### 5.2 Large Source-Code Models

LLMs demonstrate promising results on many code-related tasks, such as code generation (Chen et al., ; Liu et al., ; Chen et al., ; Le et al., ; Yue et al., ; Chen et al., ; Nijkamp et al., ; Fried et al., ; Rozière et al., ; Guo et al., ; Lozhkov et al., ; Hui et al., ), bug fixing (Jiang et al., ; Xia et al., ) and vulnerability fixing (Wu et al., ; Steenhoek et al., ; He & Vechev, ). The advances in using LLMs are attributed to the knowledge learned from massive source code and natural language text in their training datasets (Touvron et al., ; OpenAI, ). Nova is designed and trained for assembly, which has unique challenges such as low information density and diverse optimization.

### 5.3 Attention Mechanism

Standard self-attention is widely used in transformer architecture (Vaswani et al., ) to capture soft dependencies between tokens in the input. Many special attention mechanisms have been designed for better learning in various scenarios (Yang et al., ; Huang et al., ). LongCoder (Guo et al., ) combines window attention and global attention (attention sink (Xiao et al., )) to handle long input of source code. We have shown that LongCoder’s window attention is less effective than Nova’s on assembly code.

CAST (Shi et al., ) is a new neural architecture that splits the abstract syntax tree (AST) of source code into subtrees, encodes the subtrees, and aggregates to the final encoding. PA-former (Chai & Li, ) is a new neural architecture that constructs source code as pyramid input based on their AST structure and contains a pyramid attention mechanism to calculate the features in a hierarchical way. HierarchyNet (Nguyen et al., ) is a neural architecture that considers source code AST, data flow, and control flow graphs. Similarly, it cannot be applied to assembly code. Different from CAST, PA-former, and HierarchiyNet, Nova’s attention design is for assembly code, is more lightweight and can be plugged into any pre-trained generative LLM.

## 6 Limitations

One limitation is that Nova is X86-specific, as we only collect X86 assembly corpus for pre-training. This design choice is mainly affected by two facts: (1) X86 assembly is used and explored in a wide range of binary tasks (Wang et al., ; Su et al., ; Xu et al., ; Chen et al., ) compared to other assembly languages, and (2) computation limitations. However, the proposed techniques are independent of X86 assembly. Low information density and compiler optimization are the common challenges of most assembly languages such as X86, ARM, and MIPS. The proposed techniques can be applied to the future development of multi-lingual assembly LLMs.

Another potential limitation is the scale of models. We develop Nova-1B and Nova-6B. These two sized LLMs show impressive ability in assembly code decompilation and encoding. There might be potential benefit of developing larger Nova models. However, due to the computing resources limitation, we are unable to explore that in this work.

## 7 Conclusion

This work develops Nova, a generative foundation LLM for assembly code, which incorporates two key novelties (hierarchical attention and contrastive learning objectives) to address the unique challenges of assembly code.
Evaluation on downstream tasks shows the effectiveness of Nova, which outperforms existing techniques on binary code decompilation by up to 146.54% and outperforms
the latest binary code similarity detection techniques by up to 6.17%. We expect our hierarchical attention and contrastive learning techniques to benefit source code and natural language foundation models, which remains as future work.

## ACKNOWLEDGMENTS

This research was supported in part by NSF 1901242 and 2006688 and a CFI fund.

This paper was prepared for informational purposes in part by the CDAO group of JPMorgan Chase & Co and its affiliates (”J.P. Morgan”) and is not a product of the Research Department of J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.

## Appendix A Appendix

### A.1 Data Collection

This section provides additional details of the data collection. To collect assemblies from The-Stack, we attempt to compile 4 million C programs, of which 138.8K is compiled successfully. We do not collect more due to the computation resource limitations.

For the 757.1K and 138.8K source code that successfully compiled into executables (using gcc) from AnghaBench and The-Stack, we disassemble them using objdump. objdump was not able to successfully disassemble all the executables, resulting in some empty assembly code. Thus, the number of O0 – O1 we obtain from each corpus is different and smaller than the number of source codes as shown in Table 2.

Figure 6 shows an example of preprocessing the raw assembly code as described in Section 2.1.

Figure: Figure 6: Example of assembly code preprocessing
Refer to caption: x6.png

### A.2 Training Details

This section provides additional details of training. We pre-train Nova starting from DeepSeek-Coder, using the language modeling objective ($L_{lm}$) for one epoch on the AnghaBench and The-Stack corpora. This step uses a batch size of 128, with the input truncated by a 1,024 tokens limit. The model weights are updated using the AdamW optimizer. The learning rate is $5e^{-5}$, using 1000 steps of warm-up and a cosine decay to adjust the learning rate.

Then, the model is further pre-trained with the combination of language modeling and contrastive learning objectives ($L=L_{lm}+\lambda(L_{fcl}+L_{ocl})$), with $\lambda$ set to 0.1. To train with the functionality contrastive learning objective, we discard any source code that misses any one of O0 – O3 assemblies and also discard the source code whose O2 assembly is the same as its O3 assembly. As a result, this step is only trained for 0.36M data samples for one epoch. The batch size is 64, with the input truncated by a 1,024 tokens limit. The learning rate is $2e^{-5}$ using the AdamW optimizer.

The fine-tuning of both BCD and BCSD uses a batch size of 64, with the input truncated by a 2,048 token limit. Similarly, the learning rate is $2e^{-5}$ using the AdamW optimizer, and the model is fine-tuned for one epoch. During the training using the contrastive learning objectives, and the fine-tuning of BCSD, we use the average of 
 tokens’ last layer hidden states to represent the embedding of a binary function.

Infrastructure The training are conducted on eight NVIDIA RTX A100 GPUs, each with 40GB memory. Our implementation is based on Huggingface’s implementation of DeepSeek-Coder (^1^11[https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base)), PyTorch (^2^22[https://pytorch.org/get-started/locally/](https://pytorch.org/get-started/locally/)), and DeepSpeed (^3^33[https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)).

**Table 10: Comparison with applying hierarchical attention on all attention heads, using 1B models.**
| Techniques | Pass@1 | Pass@10 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O0 | O1 | O2 | O3 | Avg. | O0 | O1 | O2 | O3 | Avg. |  |
| Nova-HA | 30.58 | 19.88 | 20.58 | 16.40 | 21.86 | 44.75 | 33.13 | 33.31 | 29.82 | 35.25 |
| Nova (hierarchical on all heads) | 32.38 | 18.87 | 20.56 | 16.34 | 22.04 | 45.95 | 32.19 | 32.78 | 29.01 | 34.98 |
| Nova | 37.53 | 21.71 | 22.68 | 18.75 | 25.17 | 49.38 | 34.84 | 36.95 | 32.03 | 38.30 |

### A.3 Applying Hierarchical Attention on Half Attention Heads

The hierarchical attention mask is always applied on half of the attention heads at each layer in Nova. This ensures the LLM balances the hierarchical knowledge of assembly code and pre-trained knowledge learned by full self-attention.

Figure: Figure 7: Nova-1B correctly decompiles HumanEval-Decompile task 0.
Refer to caption: x7.png

We conducted experiments applying hierarchical attention to all the attention heads. Results in Table 10 show that
when applying hierarchical attention to all the attention heads of transformer layers, the performance does not improve enough and even drops under some settings. This implies that the standard full self-attention mechanism indeed learns knowledge that may not be captured by hierarchical attention. Thus, we only apply hierarchical attention to half of the attention heads in each transformer layer of Nova to balance the knowledge learned by standard and hierarchical attention.

Figure: Figure 8: Nova-1B correctly decompiles HumanEval-Decompile task 20.
Refer to caption: x8.png

**Table 11: Ablation study with $K=50$.**
| Benchmarks<br>Nova-CL-HA<br>Nova-HA<br>Nova-1B<br>Binutils<br>0.86<br>0.88<br>0.87<br>Curl<br>0.84<br>0.87<br>0.89<br>ImageMagick<br>0.79<br>0.80<br>0.86<br>SQLite<br>0.80<br>0.83<br>0.77<br>OpenSSL<br>0.90<br>0.92<br>0.90<br>Putty<br>0.68<br>0.66<br>0.72<br>Avg.<br>0.81<br>0.83<br>0.84<br>Table 11: Ablation study with $K=50$. | Benchmarks<br>Nova-CL-HA<br>Nova-HA<br>Nova-1B<br>Binutils<br>0.80<br>0.82<br>0.79<br>Curl<br>0.84<br>0.84<br>0.86<br>ImageMagick<br>0.70<br>0.72<br>0.79<br>SQLite<br>0.74<br>0.78<br>0.73<br>OpenSSL<br>0.89<br>0.89<br>0.88<br>Putty<br>0.59<br>0.60<br>0.65<br>Avg.<br>0.76<br>0.78<br>0.78<br>Table 12: Ablation study with $K=100$. |
| --- | --- |
| Benchmarks<br>Nova-CL-HA<br>Nova-HA<br>Nova-1B<br>Binutils<br>0.71<br>0.74<br>0.73<br>Curl<br>0.80<br>0.73<br>0.83<br>ImageMagick<br>0.61<br>0.63<br>0.73<br>SQLite<br>0.68<br>0.71<br>0.68<br>OpenSSL<br>0.85<br>0.87<br>0.84<br>Putty<br>0.53<br>0.53<br>0.55<br>Avg.<br>0.70<br>0.70<br>0.73<br>Table 13: Ablation study with $K=200$. | Benchmarks<br>Nova-CL-HA<br>Nova-HA<br>Nova-1B<br>Binutils<br>0.62<br>0.65<br>0.65<br>Curl<br>0.67<br>0.71<br>0.73<br>ImageMagick<br>0.46<br>0.51<br>0.61<br>SQLite<br>0.61<br>0.62<br>0.59<br>OpenSSL<br>0.77<br>0.79<br>0.78<br>Putty<br>0.46<br>0.46<br>0.47<br>Avg.<br>0.60<br>0.62<br>0.64<br>Table 14: Ablation study with $K=500$. |

### A.4 Binary Code Decompilation Case Studies

Figure 7 shows an example from HumanEval-Decompile (task_id 0). Given the O1-optimized assembly code, GPT-4 fails to figure out the number of function arguments correctly, missing one important argument “float e”, and thus produces wrong functionality in the decompiled code. LLM4Decompile-1B makes similar mistakes and also misses the inner nested for loop. Nova-1B correctly decompiles the assembly into source code, where the ground truth is checking if any two elements in the given list *a (with size n) are close to each other than a given threshold e.

Figure 8 shows another more complex example, HumanEval-Decompile task_id 20. Nova-1B correctly decompiles the source code, successfully figuring that the function is trying to find the two elements that are closest to each other in the given array *a, with the result stored in minmax.

Figure: Figure 9: PCA of embeddings calculated by Nova-CL-HA, Nova-HA, and Nova.
Refer to caption: x9.png

Figure: Figure 10: Comparison of attention distribution among standard and hierarchical heads.
Refer to caption: extracted/5944140/Styles/figures/attention_final_layer.png

### A.5 Binary Code Similarity Detection Ablation Study

Table 14, 14, 14, 14 show the detailed ablation study results of BCSD. Nova wins on the most benchmarks when $K=100$ or $500$, and ties with Nova-HA when $K=50$, or $200$.

### A.6 Additional Analysis of Embedding

Figure 9 shows the results of PCA of embeddings provided by Nova-CL-HA, Nova-HA, and Nova, on randomly sampled seven examples, where Nova’s embeddings are consistently more distinguishable by functionalities.

### A.7 Additional Analysis of Attention

Figure: Figure 11: Learned per-instruction soft attention observed in the lower layers
Refer to caption: extracted/5944140/Styles/figures/attention_first_layer.png

Figure 10 shows the visualizations of attention weights in the final transformer layer of two select heads with standard attention and two heads with learned hierarchical attention. Standard attention exhibits two typical patterns, namely diagonal attention (i.e. tokens attending to themselves or nearby tokens, shown in Figure 10 (a)), and broad attention (i.e. a single token attending broadly to the entire sequence, shown in Figure 10 (b)). In contrast, in Nova’s hierarchical attention, attention weights are allocated among distinct segments, each corresponding to an instruction (shown in Figure 10 (c)), that focus on tokens comprising that instruction (e.g. opcodes and operands, shown in Figure 10 (d), attentions are paid to “push”, “mov”, etc.).

Quantitatively, we have determined broad attention accounts for as much as 30% of all attention in standard heads, especially in layers 1-8 (consistent with the findings of (Clark et al., )), whereas in Nova’s hierarchical attention, no more than 5% or all attention is allocated to each instruction segment. This validates our goal of learning instruction-aware hierarchical attention in Nova.

In addition, in lower layers, we have observed attention weights to be softly distributed among tokens comprising each instruction (Figure 11), which suggests Nova initially models cross-relations among operation codes and operands in the first few layers, and later pools their summary representation into the 
 token in the later layers.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why do the evaluation for code similarity detection use cosine similarity (l. 321) when the objective (l. 212) uses the l2 distance?
2. What is the underlying metric for the Pass@k in the decompilation evaluation? Exact match, or some more lenient equivalent? It seems wrong to use exact match when, for instance, variable names would be arbitrary.
3. In Table 4, the second row is exactly the "Nova-1B" row of Table 2, but I was under the impression that "Nova-1B" was more than just "DeepSeekCoder + Nova's attention", in particular the additional training data, and CL objective. Are the numbers off, or the caption, or did I miss something?
4. When creating the assembly datasets (Appendix A.1), why go all the way to compiling executables, then using `objdump` for disassembling, with the associated possibilities of failure, rather than dump the assembly in the first place with `gcc -S`?
5. Do you have preliminary results, citations, or intuition behind the "normalizing" step of the assembly language performed in Fig. 6, in particular the addition of spaces? Is that necessary?

Minor points:
1. In the numerator on l. 265, is $f_j^q$ supposed to be $f^q$? or $f_j^p$ for which the substitution wouldn't apply?
2. l. 300, how many samples do the GPT models perform, then, to be able to compute the Pass@10 in Table 2?

Edit after discussion and update
--------------------------------------------
The overall clarity has improved, and additional information was provided.
Most questions have been answered, so I'm raising my score.
soundness: 3
strengths: Originality
--------------
1. While hierarchical attention mechanisms are not new, the design of this one is innovative in that: it takes into account the specific format and constraints of assembly instructions, and it accommodates for using regular tokens in the same sequence (e.g., natural text instructions).
2. The contrastive objective losses, as well, encode a priori knowledge of the underlying data: compilation stages preserve semantics, and optimization stages are sequential.

Quality
----------
The different contributions are overall sensible, and contribute to the performance of the model. Experiments are adequately designed, and support the conclusions of the paper. The additional experiments help understand the role of the different contributions, in particular their effect on how embeddings get clustered and the effect it can have on the final model's performance.

Clarity
---------
1. The paper includes most of the relevant information, either in the main text or appendix. Relevant literature is mentioned and cited.
2. Figures and examples make it much easier to understand the logic, especially Fig. 3.

Significance
-----------------
1. This work shows a significant improvement on benchmarks, sustained across model sizes, and adaptable to other models. This is an advancement on an important, developing field of machine learning applications.
2. Given that these improvements do not require any in-depth change (e.g., to the vocabulary) and are compatible with already pre-trained model make it easier to experiment with in different settings.
summary: The paper presents a way of training an LLM to improve its performance on tasks that require understanding of assembly code, in particular code decompilation, and assembly code similarity detection.

This is achieved by several contributions:
1. A multi-way, parallel corpus of programs written in C, as well as the corresponding assembly produced by `gcc` with different levels of optimization (0 to 3), used for further training of pre-trained LLMs.
2. A hierarchical attention mechanism, structured to summarize the content of each instruction into the representation of a single token. This mechanism is compatible with existing models.
3. Two auxiliary contrastive loss objectives: a "functionality" one that minimizes the distance between representations of the same original code, while maximizing the distance between representations of different code pieces, and an "optimization" one encoding the fact that further levels of optimization should increase the distance between program representations.

Two variants (with 1B and 6B parameters respectively) of a model trained with these changes, and further fine-tuned for the task of interest, show a large improvement over state-of-the-art.
weaknesses: Quality
----------
1. One of the 3 motivating cases in the introduction, malware detection, is not evaluated or considered at all in the rest of the paper. I understand the scope of the paper needs to end somewhere, but it would have strengthened the paper to include experiments on such a dataset.
2. Details are missing in how the authors are certain that test data sets (both for decompilation and for similarity detection) do not overlap with any of the training data, including the pre-training data of DeepSeek-Coder, even inadvertently.
3. An additional ablation on $\textrm{Nova}_{-CL}$ would have helped see if there are any non-linear interactions between HA and CL.

Clarity
---------
The overall organization of the paper could be improved. Many times, a concept, method, or setting is used in context before being formally explained. For instance:
1. If the "Related Work" section is positioned earlier, it would help introduce the baseline models (DeepSeekCoder, LLM4Decompile) that are used in the previous "Results" section, as well as attention mechanisms, including LongCoder's, also used earlier.
2. When describing the new datasets, it should be clear much earlier that "source code" really means "C code" (in the caption of Table 1, for instance), "assembly" is X86 assembly (or maybe X86-64? that's not so clear), that only `gcc` is considered as a compiler, and whether each "program" actually means a full executable program, or if it includes functions as well.
3. Similarly, the contrastive losses mention "the embedding" of a function, which is quite ambiguous in transformers, especially if the model family (encoder-decoder?) is not mentioned.
4. There is also a lot of ambiguity in notation, or the semantics of different objects. For instance:
    * Do Table 1, and Appendix A.2, refer to the original "AnghaBench" and "The-Stack" datasets, or the new datasets constructed by the authors in Section 2.1? Maybe it would be better to name the new ones.
    * In Functionality CL, l. 208 says it "optimizes Nova with the constraint", but a constraint is not a loss or objective. l. 215, "constraints can be trained" do not really make sense. It's also not obvious how the loss defined at l. 220 actually implements (a relaxation of) these constraints. It's also not explained if the sum over $f_i \in F$ is actually done over all the million embeddings in the corpus, or how it's implemented in practice.
    * K is introduced in Section 2.5, and then in 3.3., but we don't know what kinds of values will be used in practice. Also, Table 2 uses "Pass@K", but that's not the same K.
    * In captions of Fig. 4 (b) and (d), the tables are more "designs" than "implementations"
    * In Fig. 4 (b), the 1-4 indices are unfortunate as, for instance, $O0_3$ reads a lot like `-oO3`
    * The equations at l. 220 and l. 266 have a really similar form, but the use of indices $i$ and $j$ is swapped between the two, making it a bit harder

Significance
-----------------
The results are somewhat limited by the use of a single assembly language, and a single compiler, but this is acknowledged and does not seem like a fundamental limitation.

Minor points
-----------------
l. 461: "cauclated" -> "calculated"?

In the bibliography:
- Vaswani et al. is actually from 2017, not 2023 (though the arXiv version has had an inconsequential update in 2023), and a venue should be indicated (I'd suggest NeurIPS rather than arXiv)
- Other articles are missing a venue or source
- Several articles have incorrect capitalization in the title due to the lack of curly braces, e.g., use `{CodeT5}` to avoid it being rendered as "Codet5".

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The paper argues that preceding-instruction attention helps avoid reuse of the same register (e.g., "eax") immediately after it is used in the previous instruction. However, this motivation is questionable because it does not explain how further subsequent instructions are prevented from reusing the same register. A more straightforward solution could be achieved with inter-instruction attention, as it can attend to all previous instructions, which raises the concern of functional overlap between preceding-instruction attention and inter-instruction attention, thus potentially making the preceding-instruction attention redundant.
2. While Nova-1B and Nova-6B are much larger than CodeART, their performance gains in BCSD are limited. For example, in the k=100 case, CodeART sees a 17% improvement over JTrans with attention regularization, but Nova's improvement is only marginal, from 0.76 to 0.78 (as shown in Table 12). This suggests that adding hierarchical attention and other inductive biases provides limited benefits when scaling the model, and Tables 11-14 show that removing hierarchical attention does not lead to significant performance drops, questioning its overall necessity. And also in Table 5, the improvement brought by contrastive learning is much higher than the Hierarchical Attention.
3. In the paper's analysis of attention distribution (Figure 10), the standard attention frequently converges on the first token, a phenomenon known as attention sink [1]. This behavior is also evident in the analysis of hierarchical attention (Figure 10(c, d)), where each token strongly attends to the first token within its attention mask, specifically the [INST-(x-1)] token, which represents the summary of the previous instruction. But it is not common when human try to interpret the functionality of each individual instruction. Furthermore, the justification for the Hierarchical Attention Mechanism —which selectively uses specific attention heads to represent the best attention maps—is somewhat ad hoc and lacks a clearer rationale.    
4. The Hierarchical Attention Mechanism introduced in this paper represents a strong inductive bias; however, the underlying insights behind this inductive bias are not clearly explained. Additionally, the mechanism bears a striking resemblance to the Attention Regularization used in CodeART, with the primary difference being the absence of Preceding-Instruction Attention in CodeART. The effectiveness of this additional attention component has also been called into question earlier in the reivew, casting some doubt on its true contribution to the overall performance.
5. While the use of contrastive learning aligns well with the BCSD task—improving performance by ensuring that functionally similar binaries, even across different optimizations, are represented similarly—it's less clear how this objective enhances the model's ability in decompliation. The training goal focuses on increasing the similarity of tokens from the same function but compiled with different optimization settings. However, this doesn't seem directly aligned with the ultimate goal of recovering executable source code, which requires more precise structural and semantic understanding beyond just token similarity across optimization levels. It would be greatly appreciated if the authors could provide some intuition as to why this approach can lead to improvements in decompliation.
6. The authors introduced a novel optimization contrastive learning approach for the BCSD task, which had not been previously applied in the previous works, which commonly use the InfoNCE loss (line 220) or the triplet loss. As it is not discussed with deeper detail in the paper, it raises the question of whether these gains are substantial enough to justify the added complexity and whether this approach could be effectively generalized to improve other models in BCSD tasks.

[1]: Efficient Streaming Language Models with Attention Sinks
soundness: 3
strengths: 1. This paper is well-structured and easy to follow. Concepts such as hierarchical attention and contrastive learning are clearly explained.
2. The paper proposes a new method for encoding assembly code by using a Hierarchical Attention Mechanism to effectively capture the semantics of assembly instructions, while employing Contrastive Learning to ensure that functionally equivalent assembly code, even at different optimization levels, is represented similarly. This novel combination allows the model to robustly understand and learn from diverse assembly code structures.
3. The paper conducts a broad range of experiments across multiple tasks and datasets, providing comprehensive evidence of the model’s effectiveness.
4. Despite its specialized focus on assembly code, Nova's hierarchical attention is compatible with standard self-attention mechanisms, allowing it to seamlessly integrate and benefit from advancements in base models and code generation models.
summary: This paper presents Nova, a generative language model specifically crafted for understanding and generating assembly code. Nova integrates hierarchical attention mechanisms with contrastive learning to effectively capture the semantics of code. The hierarchical attention mechanism focuses on intra-instruction, preceding-instruction, and inter-instruction relations, while contrastive learning ensures that functionally equivalent code, even with different optimizations, is similarly represented. The model is evaluated on two key tasks: decompilation (recovering high-level source code from assembly) and binary code similarity detection (BCSD) (measuring the similarity between binary code functions). Nova shows superior performance in both tasks, excelling in decompilation by accurately generating source code from optimized assembly, and achieving high recall in BCSD by effectively identifying similar code across different optimization levels.
weaknesses: 1. Unclear motivation for introducing several inductive bias by Hierarchical Attention Mechanism. While the added attention mask inductive bias shows promising results in the BCD task, its impact in the BCSD task is minimal. This discrepancy raises questions about why the inductive bias performs well in one task but fails to offer significant improvements in the other.
2. Lack of Design Discussion. The paper lacks sufficient discussion on key design components like Preceding-Instruction Attention and Optimization Contrastive Learning (CL). Without Preceding-Instruction Attention, the attention design is quite similar to CodeART, raising questions about the novelty and contribution of the approach.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: For binary similarity detection, compilers may inline functions or eliminate them altogether. How does your approach handle such scenarios?

If additional information (e.g., execution traces) were provided to GPT, or if iterative interaction with GPT were allowed, could the proposed approach still outperform a GPT-based model?
soundness: 3
strengths: 1. Clear Writing and Novel Application: The paper is well-written and easy to follow. The idea of applying hierarchical attention to assembly code is interesting and novel. While hierarchical attention is commonly used in NLP tasks, applying this mechanism to assembly code is, to the best of my knowledge, unprecedented.

2. Promising Results: The evaluation results are promising. Nova demonstrates substantial improvements in both decompilation accuracy and similarity detection compared to existing models, validating its approach with strong experimental evidence.
summary: This paper presents Nova, a generative language model specifically designed for assembly code, addressing unique challenges posed by the low information density and diversity in assembly syntax due to compiler optimizations. Nova introduces a hierarchical attention mechanism and employs contrastive learning to improve the model's understanding of assembly semantics across diverse optimization levels. Trained on a large assembly corpus, Nova outperforms existing techniques in tasks like binary code decompilation and binary code similarity detection, showing  improvements in Pass@1 and Recall@1 rates over state-of-the-art models.
weaknesses: Generalizability: The model is trained exclusively on x86 assembly code, which may limit its generalizability to other assembly languages, such as ARM or MIPS.

Realism of Evaluation Settings:

(1) The decompilation prompt requires optimization level information, but it is unclear if this information is accessible in stripped binaries.

(2) For baseline models like GPT, fine-tuning with additional data isn’t necessary, raising questions about the fairness of the comparison. If GPT were given a few-shot learning setup or fine-tuned using OpenAI’s API, could it still be outperformed by the proposed approach?


Related Work: The paper omits discussion of several relevant works, which could provide a broader context for its contributions.

[1] Debin: Predicting Debug Information in Stripped Binaries. CCS 2018

[2] {DIRE}: A Neural Approach to Decompiled Identifier Renaming. ASE 2019

[3] Learning to Reverse DNNs from AI Programs Automatically. IJCAI 2022

[4] Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization. S&P 2019

[5] Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection. CCS 2017.

[6] ecompiling x86 Deep Neural Network Executables. Security 2023.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please check my concerns in the weakness section.
soundness: 3
strengths: Strengths:
+ The topic is interesting and important, addressing large language model (LLM) comprehension of assembly code.
+ The paper is well-structured and easy to follow.
summary: This paper introduces a generative model, Nova, tailored for assembly code tasks. Nova employs a hierarchical attention mechanism and is trained using contrastive learning objectives. This paper evaluates its effectiveness on two assembly code tasks.
weaknesses: Weaknesses:
- Comparison may be unfair due to different fine-tuning practices.
- Evaluation of individual components is insufficient.
- Generalization assessment is lacking.
 
(1) Unfair Comparison: Nova is evaluated on two tasks, with fine-tuning applied specifically for each. However, the baseline models (such as Table 2) do not undergo the same fine-tuning for the tasks, leading to a potentially unfair comparison.
 
(2) Component Evaluation: Nova’s hierarchical self-attention mechanism consists of three components, yet the paper lacks detailed performance assessments for each part. Despite a reasonable design, their individual impact remains unexamined.
 
(3) Contrastive Learning Objectives: The contrastive learning objectives contain two distinct components. Further evidence is necessary to substantiate the utility of each objective. Additionally, the contrastive learning approach depends on the available optimization levels. Handling unseen optimization levels at inference should be discussed.
 
(4) Normalization Process: In the data collection section, a normalization step is applied, but its relevance or benefit to Nova’s training is unclear.
 
(5) Results across different optimization levels should be explored—e.g., training on O0, O1, O2 and testing on O3.
 
(6) Random Sampling in BCSD Task: The BCSD task employs random sampling, yet statistical results are missing. Reporting such results would reduce the impact of randomness on performance claims.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. See weakness 1,2 
2. Could you provide more details about how to construct $F$ in practice used in Functional CL?
3. The authors state that hierarchical attention is only applied to half of the attention head. Since different attention heads can learn different features, I wonder if this setup is robust to the selection of the attention heads?
4. Would the pre-trained models (Nova-1B, 6B) be public available?
soundness: 2
strengths: 1. LLMs for binary code is an important topic to study
2. This work proposes new methods to train Nova based on the properties of assembly code, which is clearly motivated.
3. The proposed models show a clear improvement in binary code decompilation.
summary: This paper presents Nova, a generative LLM for assembly code. To effectively learn from assembly with low information density, it uses a novel hierarchical attention mechanism which combines intra-instruction attention, preceding-instruction attention and inter-instruction attention. It further utilizes contrastive learning to better learn the semantics of assembly code from different optimization levels. The authors demonstrate the superiority of Nova over the baselines on binary code decompilation and code similarity detection tasks.
weaknesses: 1. The comparison on code similarity detection may not be fair. For example, CodeArt uses 12 transformer blocks with 768 hidden dimensions, whose size is smaller than Nova-1B. The authors should compare Nova with the baseline under a similar size with the same pre-training data to demonstrate the superiority of Nova on code similarity detection. For the current result, we can find that compared with CodeArt, Nova actually does not show a significant improvement (e.g. both are 0.64 for Nova-1B under K=500). So it is in question whether Nova is indeed better for code similarity detection.
2. The experiments for Comparison with Techniques Handling Long Input are confusing. Specifically, it has the following problems:

a) What is "Nova’s Fine-Tuning" in Table 3? It seems Nova does not have something special in terms of fine-tuning. Does it just mean fine-tuning with hierarchical attention or also with Nova's pretraining as suggested in Line 360? 

b) What is the average token length for downstream tasks before truncation? The authors want to claim Nova is better at solving long input challenges. But I see from the Appendix that Nova uses the input length as 1024 tokens during pre-training and 2048 for fine-tuning. It may be hard to claim this length to be "long-context". Considering that assembly code should be much longer than source code and Granite-3B-Code-128K can handle 128K input tokens at most, have you tested in the benchmarks where the input context is longer, e.g. 8k/32k/128k?

3. The presentation of the paper can be improved. Specifically, a) Line 281 is unclear. The authors should clearly state that their pre-training contains two stages and the loss in Line 240 is used in the second stage. b) The ablation study should be separated into new subsections instead of mixing with Section 4.1 c) The equations are not numbered.

Meta-review: This paper studies using LLMs for binary code analysis. Despite many works used LLMs on source code tasks, they cannot be directly generalized to assembly code. The authors proposed an attention mechanism to capture the semantics in binary code for LLM training. Their method can outperform latest binary code similarity detection. 

After rebuttal, this paper received diverse scores including three positive scores and two negative ones. I read all comments and rebuttals, especially Reviewer hTSw who gave a score of 3. Despite there is no reply from Reviewer hTSw, I think almost all questions are addressed. I think this paper can be accepted.

Decision: Accept (Poster)