SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dragos Georgian Corlătescu
dragos.corlatescu@crowdstrike.com
Alexandru Dinu
alexandru.dinu@crowdstrike.com
Mihaela Gåman
mihaela.gaman@crowdstrike.com
Paul Sumedrea
paul.sumedrea@crowdstrike.com
In recent years there has been a shift from heuristics-based malware detection towards machine learning, which proves to be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note a remarkable scarcity of the data available for similarity-targeted research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one-sided quantification is especially dangerous in the context of detection bypass. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER — one of the largest malware classification data sets. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space. Our contribution is threefold: (1) we publish EMBERSim, an augmented version of EMBER, that includes similarity-informed tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VirusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.
Malware is employed as a cyber weapon to attack both companies as well as standalone users, having severe effects such as unauthorized access, denial of service, data corruption, and identity theft [1]. From signature-based malware detection [2], to more sophisticated machine learning (ML) techniques [3, 4], antivirus (AV) became a necessary layer of defense in this cyber-war. With unprecedented access to open-source tools that facilitate replicating existing malware, the number of similar malicious samples is constantly on the rise [5]. This poses a problem both from the perspective of Threat Researchers having to analyze more data each year, as well as for Data Scientists training ML models for malware detection. Including near-duplicate samples in new iterations of malware detectors is dangerous as it can introduce biases towards certain types of attacks. Thus, we can argue that being able to perform similarity search at scale is of utmost importance for the Cybersecurity world [6]. Given the vast amounts of data, machine learning is better equipped to hunt for patterns in the context of similarity search [7, 8, 9, 10, 11], rather than relying on heuristics [12, 13] or signatures [5, 14] as it has been the case until recently.
In this work, we propose to address the problem of similarity search in Windows Portable Executable (PE) files. Our choice of platform is driven by the vast majority of computers worldwide that use the Windows operating system [15]. Moreover, our motivation for approaching binary code similarity (BCS) is the challenging nature of this problem due to factors such as the absence of source code and binary code varying with compiler setup, architecture, or obfuscation [8, 16, 17]. We identified a scarcity of data in the space of BCS in Cybersecurity [17] and decided to address this by generating a benchmark for binary code similarity. Our work uses EMBER [18] – a large data set of PE files intended for malware detection – and a combination of automatic tagging tools and machine learning techniques as described below. Our hope is to enable future research in this area, with a focus on considering real-world complexities when training and evaluating malware similarity detectors.
Our contribution is threefold:
1. We release EMBERSim, an enhanced version of the EMBER data set, that includes similarity information and can be used for further research in the space of BCS.
2. We augment EMBERSim with automatically determined malware class, family and behavior tags. Based on these tags, we propose a generally-applicable evaluation scheme to quantify similarity in a pool of malicious and clean samples.
3. As a premiere for the Cybersecurity domain, we repurpose a malware classifier based on an ensemble of gradient-boosted trees [19] and trained on the EMBER data set, to quantify pairwise similarity [20]. We publish the relevant resources to reproduce, on any other sample set, both this similarity method as well as our malware tag scoring technique based on tag co-occurrence. Code and data at: CrowdStrike/embersim-databank.
The remainder of this paper is organized as follows. In Section 2 we position our research in the related context for binary code and pairwise-similarity detection. Section 3 develops on the data that this work is based on, with a focus on EMBERSim – the enhanced EMBER, augmented with similarity-derived tags. Section 4 introduces the tree-based method used for similarity and the tag enrichment workflow. The experimental setup and evaluation are discussed in Section 5. Finally, we conclude the paper with Section 6, giving an outlook of this research.
In this section, we put our work in perspective with a few related concepts and domains. We regard our own method with respect to the landscape of binary code similarity in general and with Cybersecurity in particular. We briefly discuss our choice of similarity method and its applicability to this domain. Finally, we inspect the literature exploiting EMBER [18] – the data set used as support – and note a few insights.
Binary code similarity (BCS) has, at its core, a comparison between two pieces of binary code [17]. Traditional approaches to BCS employ heuristics such as file hashing [12, 21, 22], graph matching [23] or alignment [24]. A sane default in the tool set of a Threat Analyst for similarity search is ssdeep [25], a form of fuzzy hashing mapping blocks of bytes in the input to a compressed textual representation, and using a custom string edit distance as the similarity function. Alternatively, machine learning is often chosen to tackle similarity in binary files, both with shallow [26], as well as with deep models [7]. Deep Learning, in particular, has been extensively used for BCS in the past years, from simpler neural networks [7, 8, 27, 28] to GNNs [6, 29, 30] and transformers [16, 31]. However, only a handful of the BCS ML-powered solutions [17] tackle the Cybersecurity domain [32, 33]. Out of these, most papers address similarity in functions [7, 27, 29] rather than whole programs [28].
With the present work, we propose to fill in some gaps in BCS for Cybersecurity, with a focus on whole programs. We differentiate ourselves from related research through our choice of method – repurposing an XGBoost [34] based malware classifier for PE files to leverage pairwise similarity based on leaf predictions (i.e. also known as proximities) [20]. The possibility of extracting pairwise similarities from tree-based ensembles is suggested in a few different works such as Rhodes et al. [20], Criminisi et al. [35], Cutler et al. [36], Qi et al. [37]. Similarity, in this case, is determined by counting the number of times when two given samples end up in the same leaf node, as part of prediction [36]. Intriguingly, despite its effectiveness in Medicine [38, 39] and Biology [37], we did not find any correspondence of the aforementioned method in the literature for BCS, especially applied in Cybersecurity.
Literature shows a scarcity of data for similarity detection in PE files [17]. Most of the past works are validated only against one of the benign/malicious classes [40], with less than 10K [41, 42, 40] samples in the test set [17]. Thus, we propose to conduct our research on EMBER [18], a large data set (1M samples) originally intended for malware detection in PE files. Other works using EMBER as support usually focus on malware detection [43, 44, 45, 46]. Similar to our second contribution, Joyce et al. [47] use AVClass [48] to label the malicious samples in EMBER with the corresponding malware family. However, the focus in this work is on categorizing the malware, whereas we have a different end goal (i.e. similarity search) and workflow involving the aforementioned tags. To the best of our knowledge, we are the first to release an augmented version of a large scale data set with rich metadata information for BCS.
EMBER [18] is a large data set of binary files, originally intended for research in the space of malware detection. Most of the works that use EMBER as support continue the exploration following the initial purpose for which this data has been released - malware detection [43, 44, 45, 46] and classification by malware family [47]. In the present work, we enhance EMBER with similarity-derived information. Our goal is to enable further research on the subject of ML-powered binary code similarity search applied in Cybersecurity - a rather low resource area as discussed in Section 2. Through this section, we briefly describe the data used in our experiments, as well as the new metadata added to EMBER as part of our contribution.
3.1 EMBER Overview
In this research, we use the latest version of the EMBER data set (2018, feature version 2) $^{1}$ , containing 1 million samples, out of which 800K are labeled, while the remaining 200K are unlabeled. The labeled samples are further split into a training subset of size 600K and a test subset of size 200K, both subsets being perfectly balanced with respect to the label (i.e. benign/malicious). Based on the chronological order of sample appearance, a significant portion  $(95\%)$  of the samples were introduced in 2018, with the data set primarily comprising benign samples prior to that period. Additionally, it is worth noting that the division of the data set into train and test subsets follows a temporal split that clearly delimits the two subsets. Overall, the EMBER repository clearly describes the data, contains no personally identifiable information or offensive content, and provides a readily applicable and easily customizable method for the feature extraction required for training and experimenting with new models on samples known to all vendors using VirusTotal  $(\mathrm{VT})^{2}$ .
3.2 EMBERSim - Metadata Enhancements
EMBER with AVClass v2 Tags. In order to validate and enrich the EMBER metadata, we query VirusTotal, and obtain results from more than 90 AV vendors, particularly related to the detection result (malicious or benign) and, where applicable, the attributed detection name. This step is necessary in order to assess potential label shifts over time given that detections tend to fluctuate across vendors and the extent to which this can happen has not been quantified in the original EMBER data. Next, following a procedure similar to the one outlined in EMBER<sup>3</sup>, we incorporate a run of the open source AVClass<sup>4</sup> system [48, 49] on the collected VirusTotal data. One key distinction is that we use AVClass v2 [48], which was not accessible during the original publication of EMBER. First of all, with the well-defined and comprehensive taxonomy in AVClass v2 [48] we aim to obtain standardized detection names from vendors. Secondly, we make use of the new features in AVClass2 to rank the generated tags (given vendor results) and to categorize them into FAM (family), BEH (behavior), CLASS (class), and FILE (file properties). To give an idea about interesting malware tags, we list the 10 most prevalent values below:
- CLASS: grayware, downloader, virus, backdoor, worm, ransomware, spyware, miner, clicker, infector.
$^{1}$ https://github.com/elastic/ember
$^{2}$ https://virustotal.com
3https://camlis.org/2019/talks/roth
$^{4}$ https://github.com/malicialab/avclass
FAM: xtrat, zbot, installmonster, ramnit, fareit, sality, wapomi, emotet, vtflooder, ulise.
- BEH: inject, infostealer, browsermodify,-ddos, filecrypt, filemodify, autorun, killproc, hostmodify, jswebinject.
- FILE: os:windows, packed, packed:themida, msil, bundle, prolang:delphi, small, installer:nsis, prolang:visualbasic, exploit
Tag enrichment via co-occurrence. Following the tag enrichment procedure outlined in Subsection 4.2.2, we augment the list of tags in EMBER with new tags which frequently appear together, as indicated by the tag co-occurrence matrix constructed using the AVClass v2 capabilities.
Obtaining tag ranking for evaluation. We leverage the rich tag information extracted from AVClass and augment it with tag co-occurrence data, to generate the metadata for EMBERSim. Table 1 shows how the presence of tags is correlated with the original label. This ground truth metadata can be utilized to evaluate the efficacy of our proposed similarity search method described in Subsection 4.1. The approach we explored for evaluation is described in Subsection 4.2.1.
Table 1: Previous vs. current AVClass tag presence with respect to label. Red cells indicate potential disagreement among label & tag presence (e.g. benign files with tags & malicious files without tags).
prev
curr
label	missing	present	All
missing	present	missing	present
unlabelled	95,359	8,208	1,958	94,475	200,000
benign	378,792	21,208	0	0	400,000
malicious	2,542	8,891	6,264	382,303	400,000
All	476,693	38,307	8,222	476,778	1,000,000
Table 1 highlights potential disagreements between the current and previous version of AVClass attributed tags. These tag changes can occur for benign files in limited instances (less than  $5\%$  in our case) given that greyware/adware is hard to quantify as either benign or malicious and vendors can have different views on a particular sample. Another limitation which can be observed here is that AVClass cannot attribute tags to approx.  $1\%$  of malicious samples. This is due to the fact that these samples did not have additional metadata available from VT to process at the time of writing as well as at the time of the original data set creation.
4.1 Binary Code Similarity Using Leaf Predictions
As mentioned in Section 1, this work proposes viewing similarity through the lens of leaf predictions. We regard this as one of our main contributions, provided that, as far as studied literature shows, the aforementioned method based on leaf similarity was not applied before in BCS for Cybersecurity.
The similarity method outlined in this subsection is based on leaf predictions, which means that we depend on training a tree-based classifier. Although the method itself can be applied to any tree-based ensemble, our specific choice of algorithm is an XGBoost ensemble, due to its relevance in both industry in general [50] and in academic research [51]. Given our choice of data set and the domain addressed in this work, the aforementioned XGBoost model is trained to differentiate between malicious and benign PE files. The training setup and model performance do not make the object of this research and are only briefly discussed in Section 5. Relevant to our work is how we employ the binary XGBoost classifier to quantify the pairwise similarity between samples.
Given two samples,  $s_1, s_2$ , and a trained tree-based model, we compute the leaf predictions represented as two vectors  $-x_1$  corresponding to  $s_1$  and  $x_2$  corresponding to  $s_2$ . Each of the two vectors contains the indices of the terminal nodes (leaves) for each tree in the ensemble when passing the respective sample through the model, with  $T$  being the total number of trees in the ensemble. Intuitively, the vector of leaf predictions can be viewed as a hash code, where each value at position  $i$  (i.e.  $x_1^{(i)}$  and  $x_2^{(i)}$  respectively) corresponds to a specific region in the input space partitioned by the  $i$ -th tree. Therefore, we consider two samples to be similar if they follow similar paths through the trees (denoted by the indicator function), hence obtaining similar leaf predictions. For the remainder
of this paper, we refer to this approach as leaf similarity which gives a score formally, computed as:
$$
\operatorname {L e a f S i m i l a r i t y} \left(x _ {1}, x _ {2}\right) = \frac {1}{T} \cdot \sum_ {i = 1} ^ {T} \mathbb {1} \left[ x _ {1} ^ {(i)} = x _ {2} ^ {(i)} \right] \tag {1}
$$
4.2 EMBERSim Metadata Augmentation Procedure
In Section 3, we have outlined a few insights regarding the metadata enrichment performed on EMBER. The diagram in Figure 1 provides a high-level overview of the augmentation and evaluation procedures followed. The main steps are detailed in the continuation of this subsection.
Figure 1: Summary of the data set enrichment and evaluation process.
4.2.1 From VirusTotal queries to AVClass 2
As detailed in Subsection 3.2, we use the SHA256 of the samples in the original EMBER data set to query VirusTotal and get detections from the AV vendors available on VT. We then run AVClass on the collected VirusTotal data. Our primary goal is to get standardized detection names from vendors. Secondly, we obtain a ranking of the generated tags and can categorize them into malware classes (CLASS), families (FAM) or behaviors (BEH).
4.2.2 Tag enrichment via co-occurrence
The main output AVClass provides, for each sample: the SHA256 of the file, the number of detections from vendors or a NULL value in the absence of detects, and, finally, a comma-separated list of tags (i.e. of the tag kinds: FILE, BEH, FAM, CLASS, UNK). A score is also attached to each of these tags, as displayed in Listing 1.
43211f5628568ae9e25a6011e496663b584d681303a544bc40114936a10764c6 51  
FILE:os:windows|11,CLASS:worm|5,FAM:swisyn|5,FAM:reconyc|2,FILE:packed|2
Listing 1: Example of AVClass output for a sample.
Alongside individual results, AVClass can also construct a tag co-occurrence matrix, containing all pairs of tags that appear together in samples, along with co-occurrence statistics. Considering a tag pair  $(a,b)$ , AVClass provides information about: the total number of occurrences of the tag  $a$ , the total number of occurrences of the tag  $b$ , the total number of common co-occurrences, common co-occurrences relative to tag  $a$ , and common co-occurrences relative to tag  $b$ . Tag frequency information for a given sample is determined only by the total number of distinct samples it appears in, regardless of how many vendors produced this tag. This common co-occurrence frequency can be regarded as a proxy for the relevance of the tag pair, and hence we can utilize it to augment an existing tag list with new tags which frequently appear together. That is to say, if a sample originally has only a tag  $x$ , but given the co-occurrence matrix we observe that the tag pair  $(x,y)$  frequently appears together,
then we will also add tag  $y$  to the sample. This enriching mechanism is particularly important in the context of similarity search for Cybersecurity as it allows finding samples which may share several characteristics (i.e. with respect to binary file structure), although they are tagged under different (e.g. family) names - a common practice across AV vendors. The procedure we employ for enriching the EMBER data set with tag co-occurrence information is described in Algorithm 1
Algorithm 1 Tag enrichment algorithm
Input: Sample with tag metadata, tag co-occurrence matrix, co-occurrence threshold  $T$   
Output: Extra tag info for the sample  
1: prev  $\leftarrow$  previous single tag from AVClass ▷ From EMBER, 2018  
2: curr  $\leftarrow$  current tag info from AVClass 2 ▷ Mapping from tag to score  
3: res  $\leftarrow \{\}$   
4: if not prev and not curr then  
5: return {} ▷ Nothing to do, sample is most likely benign  
6: else if prev and not curr then  
7: for tag pairs (prev  $\rightarrow x$ ) with freq(x | prev)  $\geq T$  do  
8: res[prev, x]  $\leftarrow$  freq(x | prev)  
9: else if not prev and curr then  
10: for tag  $x \in$  curr of kind  $\in$  {FAM, CLASS} do  
11: for tag pairs ( $x \rightarrow y$ ) with freq(y | x)  $\geq T$  do  
12: res[x, y]  $\leftarrow$  freq(y | x)  
13: else  
14: apply both cases above  
15: return res
4.2.3 Obtaining tag ranking for evaluation
We use the rich tag information from AVClass and the co-occurrence analysis, to generate ground truth metadata that can be utilized to evaluate the BCS method based on leaf predictions described in Subsection 4.1. The focus through Subsection 4.2 is on the tag ranking method used to generate the mentioned ground truth metadata. Considering the fact that benign samples do not have tags, we employ a tag ranking procedure applied to all malware samples from the EMBER data set. This procedure takes advantage of supplementary tags acquired through co-occurrence analysis conducted in earlier stages. For understanding the remainder of this subsection, we should recall that in the raw AVClass output, each tag is accompanied by the AVClass rank score.
Given a sample and its AVClass tag list that contains multiple tags of different kinds (FAM, CLASS, BEH, FILE) with their corresponding rank score, conditioned on a tag kind we can derive a probability distribution
$$
P (x \mid \text {k i n d} = K) = \frac {\operatorname {s c o r e} (x)}{\sum_ {\text {t a g y o f k i n d}} \operatorname {s c o r e} (y)} \tag {2}
$$
We can regard this probability as a measure of confidence and agreement between multiple vendors. Further, we denote by  $\mathrm{freq}(x,y)$  the common co-occurrence frequency of tags  $x$  and  $y$ , and by
$$
\operatorname {f r e q} (x \mid y) = \frac {\operatorname {f r e q} (x , y)}{\operatorname {f r e q} (y)} \tag {3}
$$
the relative occurrence frequency of tag  $x$  relative to the occurrences of tag  $y$ . At the core of the tag ranking algorithms is the formula in Equation (4) which extends the original AVClass rank scores with co-occurrence information:
$$
\operatorname {R a n k S c o r e} (x) = P (x \mid \text {k i n d} = K) + \sum_ {y \rightarrow x} P (y \mid \text {k i n d} = K) \cdot \operatorname {f r e q} (x \mid y) \tag {4}
$$
Note that due to the scaling by the frequency, this scoring scheme will rank original tags higher than tags obtained via co-occurrence, which may be desirable given that co-occurrence information can be noisier. Algorithm 2 introduces the tag ranking procedure.
Algorithm 2 Tag ranking algorithm
Input: Sample with tag metadata, tag kind  $K$  to rank by Output: Tag ranking for the sample   
1: Compute  $P(x\mid \text{kind} = K)$  for all tags  $x\in$  metadata   
2: Initialize RankScore[x] from  $P(x\mid \text{kind} = K)$    
3: for tag pairs  $(x\to y)\in$  metadata do   
4: if kind(x) is FAM and kind(y) is  $K$  then   
5:  $\operatorname {RankScore}[y] + = P(x\mid \operatorname {kind} = K)\cdot \operatorname {freq}(y\mid x)$    
6: ranking  $\leftarrow$  sort RankScore in descending order and filter to tags of kind  $K$    
7: return ranking
5 Experiments and evaluation
For convenience, we conduct experiments on an architecture that has 64CPUs and 256GB of RAM. A minimal set of requirements to run the method proposed in this paper, consists in being able to load the data set and model in memory. Concretely, for the EMBER data set and the trained XGBoost model the memory consumption is around 27 GB of RAM. Thus, a machine with 32 GB of RAM should be enough to replicate the results in this paper.
Experimentation, in the context of the present work, implies, first of all, determining the ground truth malware tags, as detailed in Subsection 4.2. Secondly, we fine-tune an XGBoost-powered binary malware classifier until it reaches satisfactory performance - i.e. we obtained a ROC AUC score of 0.9966 on EMBER's test split, a value close to the one (i.e. 0.9991) reported in the paper introducing the 2017 batch of EMBER data. Our fine-tuning process yielded the following set of hyperparameters: max_depth = 17, eta = 0.15, n_estimators = 2048, colsample_bytree = 1. Finally, also as part of the experimentation, we utilize the leaf predictions to compute the similarity between the samples in EMBER.