EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis
Authors: Dragos Georgian Corlatescu
Alexandru Dinu
Mihaela Gaman
Paul Sumedrea
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: similarity search
leaf similarity
boosting algorithms
tree-based methods
binary files
malware
static analysis
cybersecurity

Abstract: In recent years there has been a shift from heuristics based malware detection towards machine learning, which proves to be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note a remarkable scarcity of the data available for similarity targeted research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one-sided quantification is especially dangerous in the context of detection bypass. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER — one of the largest malware classification datasets. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space. Our contribution is threefold: (1) we publish EMBERSim, an augmented version of EMBER, that includes similarity informed tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VirusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.

EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis
Dragos Georgian Corlătescu
Alexandru Dinu CrowdStrike
CrowdStrike
dragos cow..ateecu@eerrrrrrk...k.. com
alexandru. www.andru.diiuueeeo.... www.@@rowwdsrrike..... d
Mihaela Gäman CrowdStrike mihaela. gaann@croodstrikee com
Paul Sumedrea CrowdStrike paul. houl..ssunddre@@@rodsssrr......
Abstract
In recent years there has been a shift from heuristics-basee malware detection towards machine learning, which proves be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note remarkable scarcity of the data available for similarity-targeter research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one- sided quantification is especially dangerous propose address the deficiencies in the space of similarity research on binary files, starting from EMBER one the largest malware classification data sets. We enhance EMBER with similarity information as well malware class tags, to enable further research in the similarity space. Our contribution threefold: (1) we publish EMBERSim, similarity-informe
context of detection bypass. We
augmented version of EMBER, that includes tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VorusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.
Introduction
Malware is employed cyber weapon to attack both companies well standalone users, having severe effects such as unauthorized access, denial of service, data corruption, and identity theft (1]. From signature-based malware detection [2]. more sophisticated machine learning (ML) techniques [3 4. antivirus (AV) became a necessary layer of defense in this cyber-war. With unprecedented access to open-source tools that facilitate replicating existing malware, the number of similar malicious samples is constantly on the rise [5]. This poses problem both from the perspective of Threat Researchers having analyze more data each year, as well for Data Scientists training ML models for malware detection. Including near- duplicate samples in new iterations of malware detectors is dangerous as can introduce biases towards certain types of attacks. Thus, we can argue that being able to perform similarity search at scale is of utmost importance for the Cybersecurity world [6]. Given the vast amounts of data, machine learning better equipped to hunt for patterns in the context of similarity search 7 8 9 10 [1] rather than relying on heuristics [2] 13 or signatures
5 [4] as it has been the case until recently.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

In this work, we propose to address the problem of similarity search in Windows Portable Executable (PE) files. Our choice of platform is driven by the vast majority of computers worldwide that use the Windows operating system [5 Moreover, our motivation for approaching binary code similarity (BCS) is the challenging nature of this problem due to factors such as the absence of source code and binary code varying with compiler setup, architecture, or obfuscation 16 We identified a scarcity of data in the space of BCS in Cybersecurity [17] and decided to address this by generating a benchmark for binary code similarity. Our work uses EMBER [18] a large data set of PE files intended for malware detection and a combination of automatic tagging tools and machine learning techniques described below. Our hope is to enable future research in this area, with focus on considering wealdworld complexities when training and evaluating malware similarity detectors.
Our contribution is threefold:
1. We release EMBERSim, an enhanced version of the EMBER data set, that includes similarity information and can be used for further research in the space of BCS.
2. We augment EMBERSim with automatically determined malware class, family and behavior tags. Based on these tags, we propose a similarity in a pool of malicious and clean samples.
evaluation scheme to quantify
3. As a premiere for the Cybersecurity domain, we repurpose a malware classifier based on an ensemble of gradient-boossee trees [9] and trained on the EMBER data set, to quantify pairwise similarity (20). We publish the relevant resources to reproduce, on any other sample set, both this similarity method as well as our malware tag scoring technique based on tag co-occurrence. Code and data at: CrowdStrikes
The remainder of this paper is organized related context for binary code and pairwise-similarity detection. Section 3 develops on the data that this work is based on, with a focus on EMBERSim the enhanced EMBER, augmented with introduces the tree-based method used for similarity and the tag enrichment workflow. The experimental setup and evaluation are discussed in Section 5 Finally, we conclude the paper with Section
follows. In Section 2 we position our research in the
similarity-derived tags. Section 4
6 giving an outlook of this research.
2 Related Work
In this section, we put our work in perspective with few related concepts and domains. We regard our own method with respect to the landscape of binary code similarity in general and with Cybersecurity in particular. We briefly discuss our choice of similarity method and its applicability to this domain. Finally, we inspect the literature exploiting EMBER (18] the data set used as support and note a few insights.
Binary code similarity (BCS) has, at its core, a comparison between two pieces of binary code 17.
Traditional approaches to BCS employ heuristics such as file hashing I22 21 22), graph matching (23 or alignment (24). A sane default in the tool set of a Threat Analyst for similarity search is ssdeep 25 form of fuzzy hashing mapping blocks of bytes in the input to a compressed textual
representation, and using a custom string edit distance as the similarity function. Alternatively,
machine learning is often chosen to tackle similarity in binary files, both with shallow (2), as well as with deep models (7]. Deep Learning. in particular, has been extensively used for BCS in the past years, from simpler neural networks 7 8 27 28 GNNs 6
29 30 and transformers [16] 31.
However, only a handful of the BCS ML-powered solutions tackle the Cybersecurity domain Out of these, most papers address similarity functions
32
33
27 29 rather than whole
programs (88)
With the present work, we propose fill in some gaps in BCS for Cybersecurity, with focus on whole programs. We differentiate ourselves from related research through our choice of method - repurposing an XGBoost [34] based malware classifier for PE files to leverage pairwise similarity based on leaf predictions (i.e. also known as proximities) 20] The possibility of extracting pairwise similarities from tree-based ensembles is suggested in a few different works such as Rhodes et al.
20) Criminisi et al. [35], Cutler et al. [36.. Qi et al. B7]. Similarity, in this case, is determined by counting the number of times when two given samples end up in the same leaf node, as part of prediction [36]. Intriguingly, despite its effectiveness in Medicine [88 39 and Biology (37], we did not find any correspondence of the aforementioned method in the literature for BCS, especially applied in Cybersecurity.

---

Literature shows a scarcity of data for similarity detection in PE files [7]. Most of the past works are validated only against one of the benign/malicious classes 40,, with less than 10K
4
42
40
samples in the test set [17]. Thus, we propose to conduct our research on EMBER 18 large data set (IM samples) originally intended for malware detection in PE files. Other works using EMBER as support usually focus on malware detection 43 44
46). Similar to our second contribution, Joyce et al. [47] use AVClass [48] to label the malicious samples in EMBER with the corresponding malware family. However, the focus in this work is on categorizing the malware, whereas we have different end goal (i.e. similarity search) and workflow involving the aforementioned tags. To the best of our knowledge, are the first release augmented version of large scale data set with rich metadata information for BCS.
45
3 Data
EMBER 18] is large data set of binary files, originally intended for research in the space of malware detection. Most of the works that use EMBER support continue the exploration following
the initial purpose for which this data has been released - malware detection 445 44
45
46) and
classification by malware family [7]. In the present work, we enhance EMBER with similarity- derived information. Our goal is to enable further research on the subject of ML-powered binary
code similarity search applied in Cybersecurity - rather low resource area as discussed in Section
2 Through this section, we briefly describe the data used in our experiments, as well as the new metadata added EMBER as part of our contribution.
3.1 EMBER Overview
In this research, we use the latest version of the EMBER data set (2018, feature version 2 containing 1 million samples, out of which 800K are labeled, while the remaining 200K are unlabeled. The labeled samples are further split into training subset of size 600K and test subset of size 200K, both subsets being perfectly balanced with respect to the label (i.e. benign/malicious). Based on the chronological order of sample appearance, significant portion (95%) of the samples were introduced in 2018, with the data set primarily comprising benign samples prior to that period. Additionally, it is worth noting that the division of the data set into train and test subsets follows temporal split that clearly delimits the two subsets. Overall, the EMBER repository clearly describes the data, contains no personally identifiable information or offensive content, and provides readily applicable and easily customizable method for the feature extraction required for training and experimenting with new models on samples known all vendors using VirusTotal (VT
3.2 EMBERSim Metadata Enhancements
EMBER with AVClass v2 Tags. In order to validate and enrich the EMBER metadata, we query VotusTotal, and obtain results from more than 90 AV vendors, particularly related to the detection result (malicious or benign) and, where applicable, the attributed detection name. This step is necessary in order to assess potential label shifts over time given that detections tend to fluctuate across vendors and the extent to which this can happen has not been quantified in the original EMBER data. Next, following a procedure similar to the one outlined in EMBER we incorporate a run of the open source AVClas system 48 49 on the collected VirusTote data. One key distinction is that we use AVClass v2 [48), which was not accessible during the original publication of EMBER. First of all, with the well-defined and comprehensive taxonomy in AVClass v2 (4 we aim obtain standardized detection names from vendors. Secondly, we make use of the new features in AVClass2 rank the generated tags (given vendor results) and to categorize them into FAM (family), BEH (behavior), CLASS (class), and FILE (file properties). To give an idea about interesting malware tags, we list the 10 most prevalent values below:
CLASS: grayware, downloader, virus, backdoor, worm, ransomware, spyware, miner, clicker, infector.
htaps://gituu.coo//tttttcc/ hhtps://vimsstotall.cmm
e https///ammiis.oog22001 9/alks/rohh
3

---

FAM: xtrat, zbot, installmonster, ramnit, fareit, sality, wapomi, emotet, vtflooder, ulise.
BEH: inject, infostealer, browsermodiffy ddos, filecrypt, filemodify, autorun, killproc. hostmodify, jswebinect.
FILE: os:windows, packed, tacked msil, bundle, proglang:delphi, small, in- staller:nsis, proglang:visuallasic,, exploit
Tag enrichment via co-occurrence. Following the tag enrichment procedure outlined in Subsection 4.2.2 we augment the list of tags in EMBER with new tags which frequently appear together, as indicated by the tag co-occurrence matrix constructed using the AVClass v2 capabilities.
Obtaining tag ranking for evaluation. We leverage the rich tag information extracted from AVClass and augment it with tag co-occurrence data, to generate the metadata for EMBERSim. shows how the presence of tags is correlated with the original label. This ground truth metadata can be utilized to evaluate the efficacy of our proposed similarity search method described in Subsection 4.1 The approach we explored for evaluation is described in Subsection 4.2.1
Table 1: Previous vs, current AVClass tag presence with respect to label. Red cells indicate potential disagreement among label & tag presence (e.g. benign files with tags & malicious files without tags).
prev curr label
missing missing
present
All
present
missing
present
unlabelled
95,359
8,208
1,958
94,475
200,000
benign 378,792
21,208
0
400,000
malicious
2,542
8,891
6,264
382,303
400,000
All
476,693
38,307
8,222
476,778
1,000,000
Table highlights potential disagreements between the current and previous version of AVClass attributed tags. These tag changes can occur for benign files in limited instances (less than 5% in our case) given that greyware/adware is hard quantify either benign malicious and vendors can have different views on particular sample. Another limitation which can be observed here is that AVClass cannot attribute tags approx. 1% of malicious samples. This is due to the fact that these samples did not have additional metadata available from VT to process at the time of writing as well as at the time of the original data creation.
4 Method
4.1 Binary Code Similarity Using Leaf Predictions
As mentioned in Section I this work proposes viewing similarity through the lens of leaf predictions. We regard this as one of our main contributions, provided that, as far as studied literature shows, the aforementioned method based leaf similarity was not applied before in BCS for Cybersecurity.
The similarity method outlined in this subsection is based on leaf predictions, which means that depend training tree-based classifier. Although the method itself can be applied to any tree-based ensemble, our specific choice of algorithm is an XGBoost ensemble, due to its relevance in both industry in general (50) and in academic research [5]]. Given our choice of data set and the domain addressed in this work, the aforementioned XGBoost model is trained to differentiate between malicious and benign PE files. The training setup and model performance do not make the object of this research and are only briefly discussed in Section Relevant to our work is how we employ the binary XGBoost classifier to quantify the pairwise similarity between samples.
Given two samples, 81, 82, and trained tree-based model, we compute the leaf predictions rep- resented as two vectors 2i corresponding to 81 and 22 corresponding to 82. Each of the two vectors contains the indices of the terminal nodes (leaves) for each tree in the ensemble when passing the respective sample through the model, with T being the total number of trees in the ensemble. Intuitively, the vector leaf predictions can be viewed as a hash code, where each value at position i (5)
(6) (i.e. 21 and x.2 respectively) corresponds to a specific region in the input space partitioned by the i-th tree. Therefore, we consider two samples to similar if they follow similar paths through the trees (denoted by the indicator function), hence obtaining similar leaf predictions. For the remainder

---

of this paper, we refer to this approach as leaf similarity which gives a score formally, computed as:
LeafSimilarity(: x2)
Σ 101 _
T
4.2 EMBERSim Metadata Augmentation Procedure
In Section
we have outlined few insights regarding the metadata enrichment performed on EMBER. The diagram in Figure provides high-level overview the augmentation and evaluation procedures followed. The main steps are detailed in the continuation of this subsection.
3
Detections from VirusTotal
Original
tags
Tag-en:iched
Tag ranking
Evaluation Reselts
enrichment
Top-K
Tag accurmence
ariginal tage (FAM CLASS only)
estrx tags
Leuf sinclarity
mnleanee(tsppigury), tapethit))
F4
add other ce-sccurring tags w.r.t.
FAlta
Relevance nethed
the threshole
FXRiZ
Figure I: Summary of the data set enrichment and evaluation process.
4.2.1 From VirusTotal queries to AVClass 2
As detailed in Subsection 3.2 we use the SHA256 of the samples in the original EMBER data set to query VirusTotal and get detections from the AV vendors available on VT. We then run AVClass on the collected VirusTotal data. Our primary goal is to get standardized detection names from vendors. Secondly, we obtain ranking of the generated tags and can categorize them into malware classes (CLASS), families (FAM) behaviors (BEH).
4.2.2 Tag enrichment via co-occurrence
The main output AVClass provides, for each sample: the SHA256 of the file, the number of detections from vendors NULL value in the absence of detects, and, finally, comma-separated list of tags (i.e. of the tag kinds: FILE, BEH, FAM, CLASS, UNK). A score is also attached to each of these tags, displayed in Listing !
51
:ILE:os::indows 11, CLASS worm 5, FAM: swisyn wt..:rrrcrcccce...Eaaaaaaaaeee /2
Listing 1: Example of AVClass output for sample.
Alongside individual results, AVClass can also construct tag co-occurrence matrix, containing all pairs of tags that appear together in samples, along with co-occurrence statistics. Considering a tag pair (a, b), AVClass provides information about: the total number of occurrences of the tag a, the total number of occurrences of the tag b, the total number of common co- co-occurrences, common co- occurrences relative tag a, and common co-occurrences relative to tag b. Tag frequency information for given sample is determined only by the total number of distinct samples it appears in, regardless of how many vendors produced this tag. This common co-occurrence frequency can regarded as proxy for the relevance of the tag pair, and hence we can utilize it to augment an existing tag list with new tags which frequently appear together. That to say, sample originally has only a tag x, but given the co-occurrence matrix we observe that the tag pair (z, y) frequently appears together,
5

---

then we will also add tag y to the sample. This enriching mechanism is particularly important in the context of similarity search for Cybersecurity as allows finding samples which may share several characteristics (i.e. with respect to binary file structure), although they are tagged under different (e.g. family) names a common practice across AV vendors. The procedure we employ for enriching the EMBER data set with tag co-occurrence information is described in Algorithm
Algorithm 1 Tag enrichment algorithm
Input: Sample with tag metadata, tag co-oceurrence matrix, co-oceurrence threshold T
Output: Extra tag info for the sample
prev <- previous single tag from AVClass
From EMBER, 2018 >- Mapping from tag to score
2: curr + current tag info from AVClass 2
3: res {}
4: if not prev and not curr then
5: return O
D. Nothing to do, sample is most likely benign
6: else if prev and not curr then
7: for tag pairs (prev + z) with freq(1 prev) 2 T do
8: res|prev, x] < freq(æ prev)
9: else if not prev and curr then
10: for tag E curr of kind (FAM, CLASS) do for tag pairs (x - y) with freq(y x) T do res(z, y] < fyeq x)
12:
else
14:
apply both cases above
15:
return res
4.2.3
Obtaining tag ranking for evaluation
We use the rich tag information from AVClass and the co-occurrence analysis, to generate ground truth metadata that can be utilized to evaluate the BCS method based leaf predictions described Subsection I The focus through Subsection 4.2 is on the tag ranking method used to generate the mentioned ground truth metadata. Considering the fact that benign samples do not have tags, we employ a tag ranking procedure applied to all malware samples from the EMBER data set. This procedure takes advantage of supplementary tags acquired through co-occurrence analysis conducted in earlier stages. For understanding the remainder of this subsection, we should recall that in the raw AVClass output, each tag is accompanied by the AVClass rank score.
Given sample and its AVClass tag list that contains multiple tags of different kinds (FAM, CLASS, BEH, FILE) with their corresponding rank score, conditioned on tag kind we can derive probability distribution
P(x kind = K) score
score
of kind K
We can regard this probability as a measure confidence and agreement between multiple vendors. Further, we denote by freq(z, y) the common co-occurrence frequency of tags and y, and by
freq v)
freq(z) [x y) freq(u)
the relative occurrence frequency of tag relative to the occurrences of tag y. At the core of the tag ranking algorithms is the formula Equation (4) which extends the original AVClass rank scores with co-occurrence information:
RankScore(z)) P(x kind K)+ Σ P(y kind K)- freq
(4)

---

Note that due to the scaling by the frequency. this scoring scheme will rank original tags higher than tags obtained via co-occurrence, which may be desirable given that co-occurrence information can be noisier. Algorithm 2 introduces the tag ranking procedure.
Algorithm 2 Tag ranking algorithm
Input: Sample with tag metadata, tag kind K to rank by
Output: Tag ranking for the sample
1: Compute P(x kind = K) for all tags æ E metadata
2: Initialize RankScore [x] from P(x kind K)
3: for tag pairs (x + y) metadata do
4: if' kind(x) is FAM and kind(y) is K then
5: RankScorely) += P(x kind K) freq(y zz)
6: ranking + sort RankScore in descending order and filter to tags of kind K
7: return ranking
5 Experiments and evaluation
For convenience, we conduct experiments on an architecture that has 64CPUs and 256GB of RAM. A minimal set of requirements to run the method proposed in this paper, consists in being able to load the data set and model in memory. Concretely, for the EMBER data set and the trained XGBoost model the memory consumption is around 27 GB of RAM. Thus, machine with 32 GB of RAM should be enough to replicate the results in this paper.
Experimentation, in the context of the present work, implies, first of all, determining the ground truth malware tags, as detailed in Subsection 4.2 Secondly, we fine-tune an XGBoost-powered binary malware classifier until it reaches satisfactory performance - i.e. we obtained a ROC AUC score of 0.9966 on EMBER's 's test split, a value close to the one (i.e. 0.9991) reported the paper introducing the 2017 batch of EMBER data, Our fine-tuming process yielded the following set of hyperparameters: mar max_depth = 17, eta = 0.15, n. estimators = 2048, colsample commpl__btteee = 1. Finally, also as part of the experimentation, we utilize the leaf predictions compute the similarity between the samples in EMBER.
eest-experimentation, we perform two different types of evaluation, leveraging both the malicious/be- nign labels from EMBER (as discussed in Subsection [.T] and also the tag information computed via AVClass to evaluate the quality of the similarity search outlined in Subsection 5.2 An important mention is that our evaluation employs counterfactual analysis scenario we use test set including an out of time sample, the test data being collected between 2018-11 1--1 and 1018-12, while the indexed database only includes samples up until 2018-10 10. We believe that such a scenario reflects the ever changing nature of the threat landscape as is the case with the Cybersecurity domain.
real-world like setting:
5.1 Top K Selection & Evaluation Based on Malicious/Benign Labels
We select the top K most similar samples from the whole data set, for each query-sample in the test split. Given that the XGBoost classifier did should be attached to the query process. Our first evaluation scenario is based on counting the labels (malicious/benign)) that query and its most similar samples share. Table shows the results for this type of evaluation considering the top K hits, with K in [1, 10, 50, 100]. The Mean column represents the mean hits count with the same class as the needle (e.g. for best performance, the mean should be close to K and the standard deviation should be close to 0). We report ssdeep [25] results in Table In this case, we are relaxing the evaluation condition and consider a hit the label of the query matches the label of the results and the samples have non-zero ssdeep similarity. The results indicate poor overall performance, with marginally better scores for malicious samples, most likely due to the fact that such samples contain specific byte patterns. In contrast, our proposed method (3) achieves better results for both malicious and benign queries, showing that leaf similarity well-equipped to find similar entries and becomes more precise as we decrease the value of K.
the test samples during the training, no bias
7

---

Table Label homogeneity (ssdeep 25
K Benign
Malicious
All
Mean
Std
Mean
Std
Mean
Std
1
0.51
0.49
0.8
0.39
0.66
0.47
10
3.55
4.34
7.31
4.16
5.43
4.65
50
12.31
19.24
32.59
22.12
22.45
23.07
100
19.84
34.82
61
45.22
40.42
45.31
Table 3: Label homogeneity (leaf similarity)
K
Benign
Malicious
All
Mean
Std
Mean
Std
Mean
Std
1
1
0
10
9.68
1.13
9.80
0.97
9.74
1.06
50
47.10
7.34
48.40
6.17
47.75
6.81
100
92.80
16.01
96.28
13.22
94.53
14.79
5.2 Relevance@ K Evaluation
The second evaluation flavor explored requires obtaining ground truth metadata in the form of ranking FAM tags as described Subsection 4.2 Subsequently, given query sample, we check whether its tag ranking is consistent with the top-K most similar samples. The results of this evaluation scenario are displayed in Table
The underlying assumption followed here is that metadata should be preserved for similar samples. For assessing the relevance between two tag lists, we use the similarity functions listed below:
Exact match (EM) - outputs if the inputs are exactly the same, otherwise outputs 0. is the strictest comparison method, serving
lower bound for that performance.
Jaccard index (Intersection over Union, IoU) - disregards element ordering and focuses solely on the presence of items in the inputs.
Normalized edit similarity (NES)- this function, based on Damerau-Leevnshhen distance (DL) for strings [52). allows to penalize differences in rank.
The evaluation procedure, referred to as relevance K, works as follows: given a query sample and its tag ranking, as well the top-K most similar hits (each with their own tag rankings), we use the aforementioned functions to calculate the relevance between the query and each hit. This yields relevance scores for each sample. We then repeat this process for all N query samples in subset. It is important to note that a sample is missing its tag list, we consider it to benign. Therefore, if both the query and hit missing their tag lists, the relevance 100%; if the tag list is present for either the query or hit, the relevance 0%; otherwise, we apply the similarity functions described above.
For constructing the query and knowledge base subsets, we consider the following scenarios:
Counterfactual analysis (query = test, knowledge base train test). Leveraging the train-test temporal split (i.e. timestamp (test) > timestamp (train)), ) we can simulate real-world production-like scenario, where we use unseen, but labeled data (i.e. the test set) to query the knowledge base.
2. Unsupervised labeling (query unlabelled, knowledge base = train). Given the collection of unlabelled samples, the goal is to identify similar samples within the knowledge base, with the purpose of assigning provisional label tag-
4/ that both scenarios provide very good results in terms of matching queries and hits according their tags. From similarity search perspective this means that the counterfactual analysis scenario simulating production environment, our method manages to retrieve more than 95% benign similar samples for a given unseen benign query based on tag information and more than 71% relevance of malicious samples for a given malicious query respectively even for 100 hits. This also allows for high confidence unsupervised labelling as can be seen in the bottom table. We note that the production scenario is expected to be the hardest of the two given that the queries here come

---

Table 4: Evaluation results accounting for query-hit metadata relevance as given by ranking the FAM tags. The co-occurrence threshold is set to 0.9. The following percentiles are reported, in order, for: 1 1%, 10%, 50%, 95%. both cases, the query subset size 200K, evenly split between the labels.
Unsupervised Labelling
Benign
Malicious
Relevance
Top-K
Mean (Std)
Percentiles
Mean (Std)
Percentiles
0.986 (0.120)
0, 1, 1, 0.5, 0.9, 1,
0.717 (0.451)
0, 0, 1, 0, 0.1, 0.8. 0, 0.04, 0.7,
EM
10
0,975 (0.097)
0.681 (0.355)
100
0.951 (0.124)
0.34, 0.85, 1,
0.612 (0.364)
0,986 (0.120)
0, 1,
0.812 (0.349)
0, 1,
IoU
10
0.975 (0.097)
0.9, 1,
0.780 (0.295)
0, 0.3, 0.93,
100
0.951 (0.124)
0.34, 0.85, 1,
0.714 (0.321)
0, 0.15, 0.85,
0,986 (0.120)
0,
0.797 (0.353)
0,0, 1, 0, 0.9, 0, 0.14, 0.81,
NES
10
0.975 (0.097)
0.5, 1,
0.764 (0.295)
100
0.951 (0.124)
0.34, 0.85, 1,
0.697 (0.319)
Counterfactual Analysis
Benign
Malicious
Relevance
Top-K
Mean (Std)
Percentiles
Mean (Std)
Percentiles
0.996 (0.066)
1, 1, 1, 0.3, 0.9, 1,
0.960 (0.196)
0, 1,
EM
10
0.967 (0.116)
0.730 (0.325)
0, 0.9, 0.01, 0.05, 0.77,
100
0.933 (0.152)
0.2, 0.78, 1,
0.641 (0.364)
0.996 (0.066)
1,
0.982 (0.093)
0.5, 1, 1,
loU
10
0.967 (0.115)
0.3.0.9 1,
0.834 (0.235)
0.1, 0.5, 0.95, 0.01, 0.25, 0.89,
100
0.933 (0.152)
0.2, 0.78,
0.759 (0.290)
0.996 (0.066)
1, 1,
0.982 (0.093)
0.5, 1, 1, 0.1, 0.48, 0.95, 0.01, 0.24, 0.88,
NES
10
0.967 (0.115)
0.3, 0.9, 1,
0.827 (0.236)
100
0.933 (0.152)
0.2, 0.78, 1,
0.752 (0.292)
Table Evaluation results for Mean Average Precision.
Unsupervised Labelling Malicious
Counterfactual Analysis Benign
Top-K
Benign
Malicious
1
0.986
0.716
0.996
0,960
10
0.985
0.753
0.985
0.895
50
0,973
0.714
0.967
0.812
100
0.966
0.698
0.957
0.783
from an out of time distribution compared to the samples the XGBoost model was trained on. For completeness, Table shows the evaluation results with Mean Average Precision (mAP) using exact match to check if two tag rank lists are equivalent (hence relevant).
6 Conclusions
Assumptions and limitations. The evaluation scheme described in Section 5 requires satisfying two underlying assumptions. Firstly, there are three factors that dictate the quality of the tag information for a sample: (1) the accuracy of VT vendor detections, (2) the tag normalisation and ranking algorithm used by AVClass and (3) the manner which the tag co-occurrence information leveraged. Secondly, we consider the lack of tags for sample to be an indicator that the sample is benign. Although VT detections can noisy due to various differences across vendors (see section 3) our strategy aims to mitigate this effect as much possible.

---

Potential negative societal impact and ethical concerns. The data set we enrich with additional manilarity-enhannee metadata contains known malware samples and has already benefited Cyberse- curity research for many years. Thus Cybersecurity vendors are well aware and capable to defend against all samples included in this data set.
Future research directions. We identified two different ideas that can drive future research in this space, while using our current results as support: model interpretability and refining the baseline proposed in this work. For the latter case, we see value in conducting a comparative study, with multiple tree-based models.
Final remarks. In the present work, we introduce EMBERSim, a large-scale data set which enriches EMBER with similarity information, with immediate application for sample labeling at two distinct granularities. To the best of our knowledge, we are the first to study the applicability of pairwise similarity, also denoted in this paper as leaf similarity, both in the context of binary code similarity, as well as with applicability to Cybersecurity in general. The aforementioned method shows promising results, our evaluation indicates (see Section 5). This allows us to empower malware analysts with comprehensive set of tags describing samples in a pool of malicious and clean samples they can leverage for similarity search at scale. We hope that the insights presented here will enable further research in a domain that would truly benefit from more results ML-powered binary code similarity in Cybersecurity.
Acknowledgments and Disclosure of Funding
We thank Patrick Crenshaw for the insightful discussions around pairwise similarity. We also thank Phil Roth for all the invaluable input and support provided for understanding the contents of the EMBER data set. We are also grateful to Alexandru Ghită and Ernest Szocs for their feedback and support throughout the research process.
References
[1] Syed Khurram Jah Rizvi, Warda Aslam, Muhammad Shahzad, Shahzad Saleem, and Muham- mad Moazam Fraz. Proud-mal: static analysis-baset. progressive framework for deep unsuper- vised malware classification of windows portable executable. Complex Intelligent Systems, pages 1-13, 2022.
[2] Internet security threat report. 2019. Accessed: 2023-06-06.
[3] Ferhat Ozgur Catak, Ahmet Faruk Yaz, Ogerta Elezaj, and Javed Ahmed. Deep learning based sequential model for malware analysis using windows exe api calls. Peerd Computer Science, e285, 2020.
[4] Edward Raff, Jared Sylvester, and Charles Nicholas. Learning the pe header, malware detection with minimal domain knowledge. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 121-132, 2017.
[5] Yong Tang, Bin Xiao, and Xicheng Lu. Signature tree generation for polymorphic worms. IEEE transactions on computers, 60(4):565 579, 2010.
[6] Guangming Liu, Xin Zhou, Jianmin Pang. Feng Yue, Wenfu Liu, and Junchao Wang. Code- former: A gnn-nested transformer model for binary code similarity detection. Electronics, 12 (7):1722, 2023.
[7] Nathaniel Lageman, Bric D Kilmer, Robert Walls, and Patrick D McDaniel. B in dnn: Resilient function matching using deep learning. In Security and Privacy in Communication Networks: 12th International Conference, pages 517-537. Springer, 2017.
[8] Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao, and Wei Zou. adiff: cross-version binary code similarity detection with dnn. In Proceedings of the 33rl ACM/EFE International Conference Automated Software Engineering, pages 667-678, 2018
10

---

[9] Steven HH Ding, Benjamin CM Fung, and Philippe Charland. Asm2vec: Boosting static repre- sentation robustness for binary clone search against code obfuscation and compiler optimization. In 2019 IEEE Symposium on Security and Privacy, pages 472-489. IEFF, 2019.
[10] Donghai Tian, Xiaoqi Jia, Rui Ma, Shuke Liu, Wenjing Liu, and Changzhen Hu. Bindeep: deep learning approach to binary code similarity detection. Expert Systems with Applications, (68:114388 14348, 2021.
[11] Andrea Marcelli, Mariano Graziano, Xabier Ugarte-Pedrent Yanick Fratantonio, Mohamad Mansouri, and Davide Balzarotti. How machine learning is solving the binary function similarity problem. 31st USENIX Security Symposium, pages 2099-21 21 16, 2022.
[12] Anitta Patience Namanya, Irfan U Awan, Jules Pagna Disso, and Muhammad Younas. Similarity hash based scoring of portable executable files for efficient malware detection in iot. Future Generation Computer Systems, 110:824-332, 2020.
[13] Georg Wicherski. pehash: A novel approach to fast malware clustering. LEET, 9:8, 2009.
[14] Andrew H Sung. Jianyun Xu, Patrick Chavez, and Srinivas Mukkamala. Static analyzer of vicious executables (save). In 20th Annual Computer Security Applications Conference, pages 326-334. IEFE, 2004.
[15] Operating system market Aperating-systee market- ssaasshaaaaeee aspx, 2023. Accessed: 2023-06-03.
share.
https: httpaa////emmaaethaarreee com/
[16] Sunwoo Ahn, Seonggwan Ahn, Hyungjoon Koo, and Yunheung Paek. Practical binary code similarity detection with bert-based transferable similarity learning. In Proceedings of the 38th Annual Computer Security Applications Conference, pages 361-374, 2022.
[17] Irfan U1 Haq and Juan Caballero. A survey of binary code similarity. ACM Computing Surveys, 54(3):1-38 2021.
[18] Hyrum S Anderson and Phil Roth. Ember: an open dataset for training static pe malware machine learning models. arXiv preprint arXiv:1804. 046677 2018.
[19] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1832, 2001.
[20] Jake S Rhodes, Adele Cutler, and Kevin R Moon. Geometry-and accuracy- cccuracy-preserving random forest proximities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[21] Thomas Dullien and Rolf Rolles. Graph-based comparison of executable objects (english version). Information and Communications Technology Security Symposium, 5(1):3, 2005.
[22] Zheng Wang, Ken Pierce, and Scott McFarling. Bmat-a binary matching tool. Feedback- Directed Optimization, 1999.
[23] Halvar Flake. Structural comparison of executable objects. In Ulrich Flegel and Michael Meier, editors, Detection of Intrusions and Malware & Vudnerability Assessment, GI SIG SIDAR Workshop, volume P-46 of LNI, pages 161-173. GI, 2004.
[24] Yaniv David and Eran Yahav. Tracelet-based code search in exccutables. ACM SIGPLAN Notices, 4496)334,,360, 2014.
[25] Jesse Komblum. Identifying almost identical files using context triggered piecewise hashing. Digital investigation, 3:91-97, 2006.
[26] Jeremy Z. Kolter and Marcus A. Maloof. Learning to detect and classify malicious executables in the wild. Journal of Machine Learning Research, 7:2721-2744, 2006.
[27] Fei Zuo, Xiaopeng Li, Patrick Young, Lannan Luo, Qiang Zeng, and Zhexin Zhang. Neural machine translation inspired binary code similarity comparison beyond function pairs. 26th Annual Network and Distributed System Security Symposium. The Internet Society, 2019.
11

---

[28] Yue Duan, Xuezixiang Li, Jinghan Wang, and Heng Yin. Deepbindiff: Learning program-wide code representations for binary difting. 27th Annual Network and Distributed System Security Symposium. The Internet Society, 2020.
[29] Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. Neural network- based graph embedding for cross plossforaooom binary code similarity detection. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 363-376, 2017.
[30] Zeping Yu, Rui Cao, Qiyi Tang. Sen Nie, Junzhou Huang, and Shi Wu. Order matters: Semantic-aware neural networks for binary code similarity detection. In Proceedings of the AAAI Conference on Artificial Intelligecee volume 34, pages 1155-1152, 2020.
[31] Hao Wang, Wenjie Qu, Gilad Katz, Wenyu Zhu, Zeyu Gao. Han Qiu, Jianwei Zhuge, and Chao Zhang. jtrans: jump-aware transformer for binary code similarity detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1-13, 2022.
[32] Danilo Bruschi, Lorenzo Martignoni, and Mattia Monga. Detecting self-mutating malware using control-flow graph matching. In Detection of Intrusions and Malware & Vulnerability Assessment: Third International Conference, pages 129 -143. Springer, 2006.
[33] Xin Hu, Tzi-cker Chiuch, and Kang G Shin. Large-scale malware indexing using function-call graphs. In Proceedings of the 16th ACM conference on Computer and communications security, pages 611 1-620, 2009.
[34] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[35] Antonio Criminisi, Jamie Shotton, Ender Konukoglu, al. Decision forests: A unified frame- work for classification, regression, density estimation, manifold learning and cemi-supervised learning. Foundations and trends@ in computer graphics and vision, 7(2-3):81-222 2012.
[36] Adele Cutler, D Richard Cutler, and John R Stevens. Random forests. Ensemble machine learning: Methods and applications, pages 157-175, 2012.
[37] Yanjun Qi, Judith Slein-Seetharaman and Ziv Bar-Joseph. Random forest similarity for protein- protein interaction prediction from multiple sources. In Biocomputing 2005, pages 531-542. World Scientific, 2005.
[38] Katherine R Gray, Paul Aljabar, Rolf A Heckemann, Alexander Hammers, Daniel Rueckert, and Alzheimer's Disease Neuroimaging Initiative. Random brassd similarity measures for multi-modal classification of alzheimer's disease. Neurolmage, 65:167-175. 2013.
[39] Katherine R Gray, Paul Aljabar, Rolf A Heckemann, Alexander Hammers, and Daniel Rueckert. Random forest-based manifold learning for classification of imaging data in dementia. In Ma- chine Learning in Medical Imaging: Second International Workshop, pages 159-166. Springer, 2011.
[40] Brian Ruttenberg, Craig Miles, Lee Kellogg, Vivek Notani, Michael Howard, Charles LeDoux, Arun Lakhotia, and Avi Pfeffer. Identifying shared software components to support mal- ware forensics. In Detection of Intrusions and Malware, and Vulnerability Assessment: 11th International Conference, pages 21-40. Springer, 2014.
[41] Christopher Kruegel, Engin Kirda, Darren Mutz, William Robertson, and Giovanni Vigna. Polymorphic worm detection using structural information of executables. In Recent Advances in Intrusion Detection: 8th International Symposium, pages 207-226. Springer, 2006.
[42] Saed Alrabaee, Paria Shirani, Lingyu Wang, and Mourad Debbabi. Fossil: resilient and efficient system for identifying foss functions in malware binaries. ACM Transactions on Privacy and Security, 21(2):1-34, 2018
12

---

[43] R Vinayakumar, Mamoun Alazab, KP Soman. Prabaharan Poornachandran, and Sitalakshmi Venkatraman. Robust intelligent malware detection using deep learning. IEEE Access, 46717-46738, 2019.
[44] Cangshuai Wu, Jiangyong Shi, Yuexiang Yang. and Wenhua Li. Enhancing machine learn- ing based malware detection model by reinforcement learning. Proceedings of the 8th International Conference Communication and Network Security, pages 74-78, 2018.
[45] Huu-Danh Pham, Tuan Dinh Le, and Thanh Nguyen Vu. Static pe malware detection using gradient boosting decision trees algorithm. In Future Data and Security Engineering: 5th International Conference, pages 228-236. Springer, 2018.
[46] Colin Galen and Robert Steele. Evaluating performance maintenance and deterioration over time of machine learning- based malware detection models on the ember pe dataset. In Seventh International Conference on Social Networks Analysis, Management and Security, pages 1-7. IEEE, 2020.
[47] Robert J. Joyce, Dev Amlani, Charles Nicholas, and Edward Raff. MOTIF: A malware reference dataset with ground truth family labels. Computer Security, 102922,,,, 2023,
[48] Silvia Sebastián and Juan Caballero. Avelass2: Massive malware tag extraction from AV labels. In Annual Computer Security Applications Conference, pages 42-53. ACM, 2020.
[49] Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. Avelass: tool for massive malware labeling. In Fabian Monrose, Marc Dacier, Gregory Blanc, and Joaquín García-Alfaro, editors, Research in Attacks, Intrusions, and Defenses 19th International Symposium, volume 9854 of Lecture Notes in Computer Science, pages 230-253. Springer, 2016.
[50] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning not all you need. Information Fusion, 81:84-90, 2022.
[51] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[52] Fred Damerau. A technique for computer detection and correction of spelling errors. Communi- cations of the ACM, 7(3):171-176, 1964.
[53] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, and Konrad Rieck. DREBIN: effective and explainable detection of android malware your pocket. In Proceeding of NDSS, 2014.
[54] John E. Moody. Fast learning in multi-renolltion hierarchies. In David S. Touretzky, editor, Proceedings of NIPS, pages 29-39, 1988.
[55] Jilei Yang. Fast treeshap: Accelerating SHAP value computation for trees. CoRR, 2021.
13

---

A
Appendix
A.1 EMBER data set
The original EMBER2017 data set (as reported in the paper introducing it) contains 1.1M samples, with the date of appearance of more than 95% of them spanning the year 2017. One year later, the authors have collected second batch of PE files, with 95% of them appearing in 2018. Our experiments follow the related work and are conducted on the most recent batch of EMBER data (from 2018, with feature version 2, referenced in the EMBER repository
Figures 2 and show the sample count and label distribution by date of appearance and subset, respectively. EMBER's authors state that the purpose of the 200K unlabelled samples is to aid further research on unsupervised or semi-supervised learning.
120k
unlabelled
110k
benign
100k
malicious
Number samples
90k
80k
70k
60k
50k
40k
30k
20k
10k
<018
2018-01
2018-02
2018-03
2018-04
2018-05
2018-06
2018-07
2018-08
2018-09
2018-10
2018-11 11
2018-12 12
Date appeared
Figure 2: Distribution of sample count per labels with respect the date of sample first appearance. The vast majority of the samples (95%) appear 2018
train
Subset
unlabelled
benign
test
malicious
Ok
100k
200k
300k
400k
500k
600k
700k
800k
Number of samples
Figure 3: Distribution of sample count per labels with respect to the data subset.
As measure of confidence the label, the work introducing EMBER notes that for each malicious sample, more than 40 vendors available in Votul yield positive detection result. In Figure g we show the correlation between the label and the number of VirusTotal detections. The resulting distribution is in line with expectations, observing slight fuzziness in the lower regions of detections, nonetheless respecting the 40-vendor threshold empirically set by the authors. We further note that the unlabelled samples appear to be evenly sampled from each detection bin.
httt::///gttabb..co/aaaatt////ss
14

---

unlabelled
benign
malicious
0.8
Fraction
0.6
0.4
0.2
(5, 10]
(10. 15]
(15. 20)
(20. 251
(25. 30]
(30. 351
(35. 40]
(40, 45]
50]
55]
(55 60)
(60. 65]
(65, 70]
(0.
(45.
(50.
VT detections
Figure 4: Distribution of VirusTotal detections per label.
A.2 AVClass2 tags
We turn attention to metadata obtained from the AVClass2 system [ for extracting, standardizing and ranking tag information from VirusTotal detections. Particularly of interest for our purposes are the malware family (FAM), class (CLASS) and behavior (BEH) tags, given that they serve ground truth for evaluating a similarity search method, under the assumption that the metadata should be consistent between query and its most similar samples. When applied all 1M samples in the EMBER2018 data set we explored, AVClass2 was able to identify 1219 unique FAM tags, 33 unique CLASS tags, and 43 unique BEH tags. To offer an intuition related the order of magnitude of the tag frequency, we show the top 20 most prevalent tags by kind in Figures 5
xtrat
zbot
installnonster
rannit
fareit
sality
wаponi
enotet
vLflooder
ulise
istartsurf
prepscran
adposhel
ekstak
sivis
spigot
ursnif
vittalia
dealply
10
2 10
3x
4x 10
Frequency
Figure 5: Top 20 most prevalent FAM tags (log scale).
httpg://ggthhh.cccmmmalllaaaaaa
15

---

grayware
oader
virus
backdoor
worn
rensomaare
spyware
miner
clicker
infector
rogueware
bot
hoax
keylogger
dialer
searcher
clipper
ddoser
rooter
10
10
10
10'
10*
10
Frequency
Figure 6: Top 20 most prevalent CLASS tags (log scale).
inject
infosteal
browsermodify
ddos
filecrypt
filemodify
autorun
killproc
hostsmodi: fy
Jswebinject
proxy
servstart
span
skypespan
lockscreen
processhijack
search
network
10
10
10
10
Frequency
Figure
Top 20 most prevalent BEH tags (log scale).
A.3
Evaluation
A.3.1 Label homogeneity
Our first evaluation of the similarity search method proposed in this work concerns the ability of preserving the label between query and its top K hits. We regard this evaluation scenario both as reasonable assumption for Cybersecurity focused similarity search in general and as a sanity check for the method proposed in this paper. We illustrate the results in Figure 8 with histograms showing the label homogeneity for malicious and benign queries, and their top K hits, with K E {10, 50, 100). We observe that the vast majority of the hits are concentrated in the bin close to K, meaning that almost all returned hits share the same label with the query, which is what we would expect for good performance.
16

---

Benign
100%
96 14
100%
55
50%
10%
8%
2.15
0.9%
8.4%
8.5%
0.4%
0.2%
0.24
0.1%
0.1%
(a) Top 10.
Malicious
Benign
100%
50%
50%
10%
5%
4.9%
1.8%
1.3%
1.11
8.6%
8.4%
0.4%
9.5%
0.2%
0.1%
(b) Top 50.
Malicious
Benign
100%
100%
50%
10%
10%
5%
2.6%
2.4%
1.5%
1.44
1.7%
8.8%
0.0%
9.5%
0.5%
0.3%
8.35%
0.1%
80
100
20
40
(c) Top 100.
Figure 8: Number of hits (out of top K) matching labels with query.
17

---

A.3.2 Relevance @ K
Our second evaluation approach implies assessing the relevance between query and the top hits as returned by the proposed similarity search method. The relevance @ K objective evaluates the consistency between two metadata lists. Specifically, we check how well the ranking of FAM tags for query is preserved in its top K hits. Results are shown in Figures and IO We used the following settings: for enriching the tag lists used to construct the tag ranking metadata, we used common tag co-occurrence threshold of 0.9; in order to penalize inversions in rank, we use the Normalized Edit Similarity function (as described the main paper). For the histograms, higher concentration towards 1.0 is better, meaning that almost all K hits are relevant with respect to the query. For the empirical CDFs, this translates to the curve shifting lower and to the right.
Counterfactual analysis (query = test, knowledge base train U test): in which we evaluate the effectiveness of the proposed similarity search method knowing the label of the query samples.
Histogram of Relevance@10
Empirical CDF
80
benign
1.0
benign
60
Percent
Fraction
0.0
10
Relevancegg10
Persent
20
40
malicious
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Relevance@@00
(a) Top 10.
Histogram Relevance@
Empirical
benign
benign malicious
80
0.6
50
0.4
20
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Relevanced
Percent
20
30
malicious
07
Relevance@:100
(b) Top 100.
Figure 9: Evaluation results for the counterfactual analysis case.
18

---

Unsupervised labelling (query unlabelled, knowledge base train): modelling the scenario where a practitioner wants label unseen samples using the similarity search method. Since the query set does not contain labels, we leverage the presence of malware tags to serve as indicator instead.
Histogram of Relevancag11
Emplrical CDF
benign
benign
Percent
60
0.2
0.0
0.4
0.6
Relevance@10
Parcent
20
melicious
0.0
0.1
0.
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Relevancag10
(a) Top 10.
Histogram Relevance@100
Empirical
80
bonign
malicious
0.8
Perbant
0.6
40
Fredzoon
0.4
0.0
0.1
0.
0.3
0.4
0.5
066
0.7
0.8
0.9
1.0
Relevance 100
30
melicious
Relevance@1100
(b) Top 100.
Figure 10: Evaluation results for the unsupervised labelling case.
A.4 Leaf Similarity Extended to other Data Sets
conduct an additional set of experiments, to address potential generalization concerns that might arise with respect to the leaf similarity approach proposed in this paper. Thus, variant of the DREBIN data set (53]. The DREBIN data set is composed of 129,013 samples, out which 123,453 clean and 5,560 malicious.
perform one-hot encodings on the categorical features the DREBIN data set, obtaining a total of 550K values. To reduce the amount of computational resources and time required to run this experiment, we apply heuristies (i.e., removing the "*" characters from urls), and then the "hashing trick" (54) to reduce the number of features down to 16,384 values. We perform a 80/20 split of the
19

---

data into train and test subset, while keeping the ratio between clean and dirty samples from the initial DREBIN corpus.
Prior to the final training, we perform hyper-parameter tuning for the XGBoost model, the best results being obtained when the learning rate is 0.05, maximum depth is 12, and number of estimators is equal to 1024. The final XGBoost model obtains ROC AUC score of 0.95 when evaluated on the test subset. Then we apply the same approach described in the paper, using leaf similarity class homogeneity check. We obtained the results displayed in Table 6 when considering the "test vs test+train" scenario.
Table 6: Label homogeneity evaluation scenario for leaf similarity over the DREBIN data set.
K
Benign Mean
Malicious
All
Std
Mean
Std
Mean
Std
0
10
9.96
0.41
8.95
2.30
9.92
0.65
50
49.75
2.19
38.51
16.42
49.27
4.61
100
99.50
4.19
71.21
35.10
98.30
10.10
As can be observed from these results, our method achieves good performance another cyberse- curity oriented dataset. This discovery makes us confident that as part of our future work we could enrich current benchmark based on EMBER with other types of malware data.
A.5
Discussion model interpretability
We briefly address model interpretability for our tree-based model, using a TreeSHAP algorithm [55]. We conduct quick experiment with the end goal of understanding the connection between the features' importance a group of samples sharing a class tag and the features' importance for the samples that were enriched with the identical tag. We apply the following steps, and compute:
the SHAP values for the samples with tag extracted from AVClass;
the ranking in terms of importance for the features extracted at the previous step;
the SHAP values as well as the feature importance ranking for the samples with tag X enriched using our method;
the Spearman correlation between the group feature ranking and the individual sample feature ranking.
Through random sampling, we obtain a moderate correlation, constantly scoring greater than 0.3 which proves that the sample with generated tags are similar from this perspective to the samples that already attached tags from AVClass. Please note that these results were obtained with tag co-occurrence threshold of 0.9.
We conclude that line can be drawn between the enriched tags and the feature importance using SHAP values, but the results are not definitive. Additional research needs to be done in the area of interpretability better understand the model outputs.
Model interpretability represents an interesting future research topic. From this angle, we perform a very narrow assessment of our similarity method, as outlined in this subsection. However, believe that this represents a separate line of research that should be the primary focus in a future study.
20

---

B
Appendix
B.1 EMBERSim data set documentation
The data used in the databank released as part of this research consists of a postprocessing of the EMBER data set, such that we can easily perform binary code similarity search over this data. The original data set consists of feature vectors from million Windows portable executable (PE) samples. The original EMBER release includes a feature extractor which provides 8 groups diverse features from each of these PE samples.
In the original formulation, the authors included malicious and benign labels, perfectly balanced among classes along with family tags derived from the first version on the AVClass package. In the most up to date version of the data set only 800K out of I million samples are tagged as being either malicious or benign respectively. We use all the conventional features from the original implementation as well as the benign and malicious labels. As part of our contribution, we enhance each sample in EMBER with iteration of the AVClass package (for which we get tags of kinds FAM, CLASS, BEH, FILE) as well as the top 100 most similar samples to given query according to our leaf similarity method. We provide an extensive evaluation of the similarity method proposed this paper, showing that ensures both label and family tag consistency. Thus, this research results in million fully tagged with malware families and labeled (i.e. benig//malicious) samples along with 400k samples for which the top 100 most similar samples are included. Please note that the most similar samples can also be deduced for the rest of the 600K samples based on the information provided. However, our focus is to keep these samples as indexed database of known samples among which we can search for similar samples.
tags extracted using both an up to date second
We propose a split designed simulate production use cases as closely as possible. We first construct a train split (identical to the EMBER labelled training corpus) on which we propose training similarity informed models. Then we break our queries into two different splits. The first split supports a validation scenario in which we query indexed database with all the samples seen during training. In this case both the queries and the indexed database span the same time intervals. The second split supports a production-lik, evaluation scenario, in which search through indexed database using queries collected in a time interval disjoint with the interval of the training subset. The splits corresponding to the aforementioned two scenarios, are published using the CSV format.
B.2
Intended uses
We hope that our proposed contribution will encourage further research in the area of binary code similarity given the large scale corpus we have put together for this purpose. Our intention is to release both our similarity informed tagging technique as well as our similarity method such that upcoming research can construct novel similarity algorithms for binary code using realistic setting.
B.3 Data set download
Instructions to download all necessary artifacts as well as reproduce our experiments are provided at: btttps://iithie cwm/ Cro.dStrikee httpstttriieeeeberiiimimmm embersin-dataaank. This public GitHub repository links to all the other required open-source resources used in this work and also contains all the code necessary transform the original data into our format as well all the sample code required to evaluate our method following the details presented in our paper.
B.4 Author responsibility for violation of rights
Our proposed contribution does not contain any kind of sensitive data, personally identifiable information or confidential information. The authors are not aware of any possible violation of rights and take responsibility for the published data.
B.5
Data set hosting and long-term preservation
The authors take full responsibility for the availability of the processed data in the provided repository. No statement can be made about the availability of third-party dependencies such as the EMBER data set or the AVClass repository. However, all the samples (binary files) part of the EMBER data set are
21

---

available for download using VirusTotal. TotusTotal is service free of charge for con-commercial use and is supported by both private companies as well public contributors as central source of truth for the cybersecurity community. We make sure to include all the tag information for these samples in our repository which is stored on Zenodo and will continue be available long as CERN exists.
B.6 License
We release our code under AGPL V3 Clause License, therefore allowing the redistribution and use in source and binary forms, with or without modification as long as the resulting code remains subject the same license. The metadata is released under Creative Commons Attribution 4.0 International Public License allowing others to freely share, copy and redistribute the material in any medium or format.
22


Reviews:
[{"review_id": "vLSlhnQkhG", "reviewer": "Reviewer_xndR", "summary": "", "questions": "", "limitations": "The authors clearly acknowledge the noisiness of AV vendor detections and the limitations of the tagging approach. They suggest interpretability analysis in future work to better understand the model, which could help address these issues. Overall, the limitations are adequately addressed.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Addresses an important problem with a novel technique and useful benchmark dataset.\n- Comprehensive metadata enrichment facilitates supervised learning and evaluation.\n- Promising results on malware similarity search in initial evaluations.\n- Clearly written and technically sound overall.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "pHUUeSHJAY", "reviewer": "Reviewer_ts2F", "summary": "", "questions": "", "limitations": "\n1. The paper primarily evaluates the proposed method on a single dataset (EMBER), which may not fully represent the diversity and complexity of real-world malware. This limits the generalizability of the authors' findings and raises questions about the effectiveness of the proposed method in real-world settings.\n\n2. Lack of interpretability. While the proposed method achieves high accuracy in detecting similar binary code, it is unclear how the tags generated by the tag enrichment workflow correspond to specific code features or how they would change when changing the XGBoost's hyperparameters.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A novel tree-based method for binary code similarity search, which has the potential to significantly improve malware detection.\n\n2. The proposed tag enrichment workflow is highly effective and improves the accuracy of the similarity search.\n\n3. Discuss the proposed method's limitations and potential future research directions. This discussion highlights the need for further research in areas such as improving the accuracy of the similarity search and addressing the challenges of dealing with obfuscated code.\n\n4. The authors provide a clear and detailed description of their experimental setup, including the dataset introduced, the evaluation metrics, and the performance of their method on various benchmarks. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TFta2F2MZ5", "reviewer": "Reviewer_qd4B", "summary": "", "questions": "", "limitations": "- The authors have largely thoughtfully addressed limitations. I think they are being fairly transparent with what they are doing early on and clearly identifying that as \"binary similarity.\" I would argue that this is a narrow-minded view, but that may be a matter of taste. I think overall at least the authors use this term consistently, and their limitations section is clear about the major limitations of the work.\n\n- I am not sold on whether class-based similarity is truly useful with respect to other methods, e.g., the heuristic or semantic methods that the authors mention. The authors do not measure against existing ground truth in the area, which to me would seem to be a limitation--perhaps it would be especially limiting if the authors were proposing this as a SOTA innovation in this space, rather than merely a dataset for other researchers to use.\n\n- Following on last point: I think the paper does little to abate the concern around its perspective within the context of the state of the art. There is a lot of implicit assumption that you are buying into this way of analyzing applications. At least for folks who do these kinds of signature and tag-based analyses, I think this is a very reasonable assumption--but it is not universal.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- I looked through their artifact, and it is well and professionally done, including notebooks which demonstrate how the evaluation was performed. The code seems sensible and above the bar (especially for research-level code).\n\n- The topic, binary similarity, is topical and very relevant. EMBER is\n  a state-of-the-art dataset in this space. It is very interesting to\n  extend it with information that relates to binary similarity, and a\n  publicly available dataset of this form could accelerate\n  state-of-the-art workloads in this area.\n\n- The paper makes a lot of interesting points and the evaluation is\n  done in a thoughtfully-narrative way that was able to be understood\n  by just reading the paper in a linear way. The evaluations also\n  report reasonable results, as far as I can tell.\n\n- The authors have been thoughtful to account for the fact that EMBER\n  has evolved over time, e.g., by using AVClass v2 tags--which were\n  not available when EMBER was released.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ztU1N7ak4Z", "reviewer": "Reviewer_6wpR", "summary": "", "questions": "", "limitations": "No negative impact was identified as the malware is not going to be executed. Although the known malware can be checked for similarity, it would be a good experiment to check for similarity with unknown binary outside the used database.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The scarcity of malicious data is a main challenge In the cybersecurity domain. The authors have extended an existing dataset with an additional metric, a similarity-based tag, which can be useful in machine learning techniques for finding similar malware.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Q6zr1ouj4j", "reviewer": "Reviewer_BNKs", "summary": "", "questions": "", "limitations": "The paper stated the limitation of their tagging process due to missing tags and tagging disagreement between the current and previous versions of AVClass. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper released an augmented dataset to address the lack of large-scale benchmarking dataset for binary code similarity research. The paper has concise presentation of the proposed procedure for tag augmentation. The paper also proposed two metrics for evaluating the performance of BCS algorithms. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper focuses on the domain of malware analysis, and extends the popular EMBER dataset to enable better research on binary similarity. The authors publish an augmented dataset based on EMBER, which includes similarity-derived tags for benchmarking malware similarity detectors. 

Reviewers agree that the paper is well written, clear and thorough, and that the authors do a good job of clearly defining and discussing its scope and limitations. 

The artifacts included with the paper are well done, and should have a real positive impact on research in this area.

The recommendation is accept.
Decision: Accept (Poster)