GenTAL: Generative Denoising Skip-gram Transformer for Unsupervised Binary Code Similarity Detection

Abstract: Binary code similarity detection serves a critical role in cybersecurity. It alleviates the huge manual effort required in the reverse engineering process for malware analysis and vulnerability detection, where often the original source code is not available for analysis. Most of the existing solutions focus on a manual feature engineering process and customized code matching algorithms that are inefficient and inaccurate. Recent deep-learning-based solutions embed the semantics of binary code into a latent space through supervised contrastive learning. However, one cannot cover all its possible forms in the training set to learn the variance of the same semantic. In this paper, we propose an unsupervised model aiming to learn the intrinsic representation of assembly code semantics. Specifically, we propose a Transformer-based auto-encoder-like language model for the low-level assembly code grammar to capture the abstract semantic representation. By coupling a Transformer encoder and a skip-gram-style loss design, it can learn a compact representation that is robust against different compilation options. We conduct experiments on four different block-level code similarity tasks. It shows that our method is more robust compared to the state-of-the-art.

Reviews and author responses:

The paper proposes an ML-based binary similarity detection technique that uses a skip-gram to encode the instruction sequences and then uses a transformer to encode the whole code fragment. The paper evaluated in cross-compilation and obfuscation settings. + The paper is well-written and described very well. 
+ The topic is important, especially for binary code analysis and reverse engineering. 


- The task of binary similarity detection is not super novel. The paper claims that they are the first to propose block-level binary similarity. However, there are many such works including InnerEye, DeepBinDiff, PalmTree, etc. The authors did not compare with any one of them. In fact, they did not even mention these papers.

-The modeling is also not novel. It is a combination of skip-gram and transformers.

- Although the paper compared with many alternate model choices, they failed to compare with any baselines.

- The authors did not evaluate how well the technique will work on sophisticated obfuscation. More interestingly, can it work on unseen obfuscation given they are using denoising skip-gram? 


 The paper works on an important topic. However, they assume a weak setting (did not use sophisticated obfuscation) and did not compare with the state-of-the-art baselines. Thus, I am not convinced with the advancement reported by the proposed technique.

==+== A. Paper summary

This paper proposes a new model architecture to encode assembly code. By using a new reconstruction loss with the new architecture, the paper achieves better fine-tuning results on downstream binary code similarity detection tasks.  
==+== B. Strengths 

The performance of the new architecture is much better than baseline MLM and its variants. 


==+== C. Weaknesses  

The novelty of this paper is not very exciting. Also, only a single subtask is given (i.e., binary similarity check), which can hardly prove the generalization of the model architecture over the assembly code.

Some important implementation details are missing.

The paper writing is quite confusing.

The evaluation is not very interesting.

Questions and concerns:

The idea of encoding the instructions together was used across multiple assembly applications (ASM2Vec, https://proceedings.neurips.cc/paper/2019/hash/093b60fd0557804c8ba0cbf1453da22f-Abstract.html ). This paper adopts this idea into transformers and masked the entire instruction for pretraining.  The idea is not very exciting.

There is a lot of typos in Sec 2.2 and Sec 3.3. I can hardly understand your derivation. For example, 1) I guessed out the meaning EM/TE/H in eq.2 and eq.3. But it never appears anywhere else and also in Figure 2.  2) I am confused about how you obtained eq.10 from eq.9. Also, what is K? What do you mean by `using an arbitrary function'? Is that a trainable matrix? These notations are broken. 

Many details are missing, e.g., your training hyperparameters for baseline MLM. This is important because pretraining a transformer using assembly code can easily run out of GPU memory.  According to your maximum sequence length ( which is 512 given in Sec.4), I don’t think this is enough for a moderate-size assembly code. Can you plot the histogram of the token sizes using your pretraining dataset? 

Also, your baseline explanation is confusing (Sec 4.1, also a lot of typos and a lot of notations that are never explained). What is IDF? You should at least explain how TFIDF works. The paper constantly mentions `MLM outperforms all previous methods and machine learning baselines’, but no experiments are given. You can try out some naive machine learning baselines (e.g, a simple bag-of-words method given in https://arxiv.org/pdf/2006.05265v6.pdf). 

Regarding your experiment results, I don’t know why O2-O3 matching is easier than O0-O0 matching. Intuitively, O0 with disabled optimization contains more information regarding the code and may be easier to identify its cross-compilation counterparts. 
 The paper has some novelty but needs a lot of improvements.

This paper proposes an enhanced transformer for dealing with the generation of assembly code. This is achieved by feeding the model with one instruction at the time (as BERT did), and hence each of these is divided into sub-words in a position-preserving way.
The authors shows the efficacy of GenTAL against many other work proposed in literature, showing that their solution is able to better grasp all the possible different level of optimization applied by the compiler (O1 to O3). Pros:
+ domain is very interesting, as assembly language is very difficult to deal with
+ good experimental section that considers many models from the literature
+ creating and learning from different compiler optimization technique is a good idea, since compilers might be very aggressive on edit they apply on the code

Cons:
- Obfuscation is lacking
- Obscure for non-BERT experts
- Writing can be improved

The domain is very interesting, since assembly is very difficult to tame. Also, the development of technologies like this could really help in many different domains, like reverse engineering (which is still a daunting task).
The issues of the paper are expressed as follows:

**Obfuscation is lacking**
This scenario is the most complex one, and it is treated poorly inside the paper. While the optimization levels are treated and described, this is entirely skipped by the authors. The readers that have not much knowledge on the domain will be lost. Also, is this obfuscation applied at assembly or at the source code level? The paper that is quoted seems to point on the second one, but the author should add these details to the paper. 

**Obscure for non-BERT experts**
The authors could have spent more time in better describing the technology, as it is very difficult to read if there is no or partial knowledge of the domain.

**Writing can be improved**
There is no section dedicated to the limitations, and there is a lot of space that could be saved to write them down (like, 4.1 is a wall of badly spaced text).  I am marginally above the acceptance, because some details were obscure to me (lack of my knowledge on the domain, and dense technical details on the paper), but I see the contribution.

Meta-review: This paper presents a transformer model for learning representations of assembly code blocks, trained using a variant of the masked language modeling objective that encodes the full code block token sequence into a single bottleneck vector and then uses that vector to decode all the masked out tokens.  Overall reviewer assessment for this paper is on the rejection side, mostly due to the not so novel model architecture and training objective.  Experiments show that this variant of the MLM performs significantly better than the standard MLM objective without the bottleneck, which surprisingly is even worse than the simple TF-IDF in many tasks.  This raises questions and it is unclear from the paper why the variant with a sequence-level bottleneck should perform better than the standard MLM.  Binary code similarity detection has many implications in security, so this is a good domain to explore more in, and I encourage the authors to continue to improve this work and send it to the next venue.

One related work also published in the ML community comes to mind that the authors might not be aware of: Graph matching networks for learning the similarity of graph structured objects by Li et al., ICML 2019, which also looked at binary code similarity detection, but works at the function level.  It would be good to also take a look for other potentially missing related work.

Decision: Reject