SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Post-experimentation, we perform two different types of evaluation, leveraging both the malicious/beign labels from EMBER (as discussed in Subsection 5.1) and also the tag information computed via AVClass to evaluate the quality of the similarity search as outlined in Subsection 5.2. An important mention is that our evaluation employs a counterfactual analysis scenario in a real-world like setting: we use a test set including an out of time sample, the test data being collected between 2018-11 and 2018-12, while the indexed database only includes samples up until 2018-10. We believe that such a scenario reflects the ever changing nature of the threat landscape as is the case with the Cybersecurity domain.
5.1 Top K Selection & Evaluation Based on Malicious/Benign Labels
We select the top K most similar samples from the whole data set, for each query-sample in the test split. Given that the XGBoost classifier did not see the test samples during the training, no bias should be attached to the query process. Our first evaluation scenario is based on counting the labels (malicious/benign) that a query and its most similar samples share. Table 3 shows the results for this type of evaluation considering the top K hits, with K in [1, 10, 50, 100]. The Mean column represents the mean hits count with the same class as the needle (e.g. for best performance, the mean should be close to K and the standard deviation should be close to 0). We report ssdeep [25] results in Table 2. In this case, we are relaxing the evaluation condition and consider a hit if the label of the query matches the label of the results and the samples have non-zero ssdeep similarity. The results indicate poor overall performance, with marginally better scores for malicious samples, most likely due to the fact that such samples contain specific byte patterns. In contrast, our proposed method (Table 3) achieves better results for both malicious and benign queries, showing that leaf similarity is well-equipped to find similar entries and becomes more precise as we decrease the value of K.
Table 2: Label homogeneity (ssdeep [25])
K	Benign	Malicious	All
Mean	Std	Mean	Std	Mean	Std
1	0.51	0.49	0.8	0.39	0.66	0.47
10	3.55	4.34	7.31	4.16	5.43	4.65
50	12.31	19.24	32.59	22.12	22.45	23.07
100	19.84	34.82	61	45.22	40.42	45.31
Table 3: Label homogeneity (leaf similarity)
K	Benign	Malicious	All
Mean	Std	Mean	Std	Mean	Std
1	1	0	1	0	1	0
10	9.68	1.13	9.80	0.97	9.74	1.06
50	47.10	7.34	48.40	6.17	47.75	6.81
100	92.80	16.01	96.28	13.22	94.53	14.79
5.2 Relevance@K Evaluation
The second evaluation flavor explored requires obtaining ground truth metadata in the form of ranking FAM tags as described in Subsection 4.2. Subsequently, given a query sample, we check whether its tag ranking is consistent with the top-K most similar samples. The results of this evaluation scenario are displayed in Table 4.
The underlying assumption followed here is that metadata should be preserved for similar samples. For assessing the relevance between two tag lists, we use the similarity functions listed below:
- Exact match (EM) – outputs 1 if the inputs are exactly the same, otherwise outputs 0. It is the strictest comparison method, serving as a lower bound for that performance.
- Jaccard index (Intersection over Union, IoU) – disregards element ordering and focuses solely on the presence of items in the inputs.
- Normalized edit similarity (NES) – this function, based on Damerau-Levenshtein distance (DL) for strings [52], allows us to penalize differences in rank.
The evaluation procedure, referred to as relevance@K, works as follows: given a query sample and its tag ranking, as well as the top-K most similar hits (each with their own tag rankings), we use the aforementioned functions to calculate the relevance between the query and each hit. This yields K relevance scores for each sample. We then repeat this process for all N query samples in a subset. It is important to note that if a sample is missing its tag list, we consider it to be benign. Therefore, if both the query and hit are missing their tag lists, the relevance is  $100\%$ ; if the tag list is present for either the query or hit, the relevance is  $0\%$ ; otherwise, we apply the similarity functions described above.
For constructing the query and knowledge base subsets, we consider the following scenarios:
1. Counterfactual analysis (query = test, knowledge base = train  $\cup$  test). Leveraging the train-test temporal split (i.e. timestamp(test) > timestamp(train)), we can simulate a real-world production-like scenario, where we use unseen, but labeled data (i.e. the test set) to query the knowledge base.
2. Unsupervised labeling (query = unlabelled, knowledge base = train). Given the collection of unlabelled samples, the goal is to identify similar samples within the knowledge base, with the purpose of assigning a provisional label or tag.
Table 4 shows that both scenarios provide very good results in terms of matching queries and hits according to their tags. From a similarity search perspective this means that in the counterfactual analysis scenario simulating a production environment, our method manages to retrieve more than  $95\%$  benign similar samples for a given unseen benign query based on tag information and more than  $71\%$  relevance of malicious samples for a given malicious query respectively even for 100 hits. This also allows for high confidence unsupervised labelling as can be seen in the bottom table. We do note that the production scenario is expected to be the hardest of the two given that the queries here come
Table 4: Evaluation results accounting for query-hit metadata relevance as given by ranking the FAM tags. The co-occurrence threshold is set to 0.9. The following percentiles are reported, in order, for:  $1\%$ ,  $10\%$ ,  $50\%$ ,  $95\%$ . In both cases, the query subset size is 200K, evenly split between the labels.
Unsupervised Labelling
Relevance	Top-K	Benign	Malicious
Mean (Std)	Percentiles	Mean (Std)	Percentiles
EM	1	0.986 (0.120)	0, 1, 1, 1	0.717 (0.451)	0, 0, 1, 1
10	0.975 (0.097)	0.5, 0.9, 1, 1	0.681 (0.355)	0, 0.1, 0.8, 1
100	0.951 (0.124)	0.34, 0.85, 1, 1	0.612 (0.364)	0, 0.04, 0.7, 1
IoU	1	0.986 (0.120)	0, 1, 1, 1	0.812 (0.349)	0, 0, 1, 1
10	0.975 (0.097)	0.5, 0.9, 1, 1	0.780 (0.295)	0, 0.3, 0.93, 1
100	0.951 (0.124)	0.34, 0.85, 1, 1	0.714 (0.321)	0, 0.15, 0.85, 1
NES	1	0.986 (0.120)	0, 1, 1, 1	0.797 (0.353)	0, 0, 1, 1
10	0.975 (0.097)	0.5, 0.9, 1, 1	0.764 (0.295)	0, 0.3, 0.9, 1
100	0.951 (0.124)	0.34, 0.85, 1, 1	0.697 (0.319)	0, 0.14, 0.81, 1
Counterfactual Analysis
Relevance	Top-K	Benign	Malicious
Mean (Std)	Percentiles	Mean (Std)	Percentiles
EM	1	0.996 (0.066)	1, 1, 1, 1	0.960 (0.196)	0, 1, 1, 1
10	0.967 (0.116)	0.3, 0.9, 1, 1	0.730 (0.325)	0, 0.1, 0.9, 1
100	0.933 (0.152)	0.2, 0.78, 1, 1	0.641 (0.364)	0.01, 0.05, 0.77, 1
IoU	1	0.996 (0.066)	1, 1, 1, 1	0.982 (0.093)	0.5, 1, 1, 1
10	0.967 (0.115)	0.3, 0.9, 1, 1	0.834 (0.235)	0.1, 0.5, 0.95, 1
100	0.933 (0.152)	0.2, 0.78, 1, 1	0.759 (0.290)	0.01, 0.25, 0.89, 1
NES	1	0.996 (0.066)	1, 1, 1, 1	0.982 (0.093)	0.5, 1, 1, 1
10	0.967 (0.115)	0.3, 0.9, 1, 1	0.827 (0.236)	0.1, 0.48, 0.95, 1
100	0.933 (0.152)	0.2, 0.78, 1, 1	0.752 (0.292)	0.01, 0.24, 0.88, 1
Table 5: Evaluation results for Mean Average Precision.
Top-K	Unsupervised Labelling	Counterfactual Analysis
Benign	Malicious	Benign	Malicious
1	0.986	0.716	0.996	0.960
10	0.985	0.753	0.985	0.895
50	0.973	0.714	0.967	0.812
100	0.966	0.698	0.957	0.783
from an out of time distribution compared to the samples the XGBoost model was trained on. For completeness, Table 5 shows the evaluation results with Mean Average Precision (mAP) using exact match to check if two tag rank lists are equivalent (hence relevant).
Assumptions and limitations. The evaluation scheme described in Section 5 requires satisfying two underlying assumptions. Firstly, there are three factors that dictate the quality of the tag information for a sample: (1) the accuracy of VT vendor detections, (2) the tag normalisation and ranking algorithm used by AVClass and (3) the manner in which the tag co-occurrence information is leveraged. Secondly, we consider the lack of tags for a sample to be an indicator that the sample is benign. Although VT detections can be noisy due to various differences across vendors (see section 3), our strategy aims to mitigate this effect as much as possible.
Potential negative societal impact and ethical concerns. The data set we enrich with additional similarity-enhancing metadata contains known malware samples and has already benefited Cybersecurity research for many years. Thus Cybersecurity vendors are well aware and capable to defend against all samples included in this data set.
Future research directions. We identified two different ideas that can drive future research in this space, while using our current results as support: model interpretability and refining the baseline proposed in this work. For the latter case, we see value in conducting a comparative study, with multiple tree-based models.
Final remarks. In the present work, we introduce EMBERSim, a large-scale data set which enriches EMBER with similarity-derived information, with immediate application for sample labeling at two distinct granularities. To the best of our knowledge, we are the first to study the applicability of pairwise similarity, also denoted in this paper as leaf similarity, both in the context of binary code similarity, as well as with applicability to Cybersecurity in general. The aforementioned method shows promising results, as our evaluation indicates (see Section 5). This allows us to empower malware analysts with a comprehensive set of tags describing samples in a pool of malicious and clean samples they can leverage for similarity search at scale. We hope that the insights presented here will enable further research in a domain that would truly benefit from more results - ML-powered binary code similarity in Cybersecurity.
We thank Patrick Crenshaw for the insightful discussions around pairwise similarity. We also thank Phil Roth for all the invaluable input and support provided for understanding the contents of the EMBER data set. We are also grateful to Alexandru Ghită and Ernest Szocs for their feedback and support throughout the research process.
[1] Syed Khurram Jah Rizvi, Warda Aslam, Muhammad Shahzad, Shahzad Saleem, and Muhammad Moazam Fraz. Proud-mal: static analysis-based progressive framework for deep unsupervised malware classification of windows portable executable. Complex & Intelligent Systems, pages 1-13, 2022.
[2] Internet security threat report. https://docs.broadcom.com/doc/istr-24-2019-en, 2019. Accessed: 2023-06-06.
[3] Ferhat Ozgur Catak, Ahmet Faruk Yazı, Ogerta Elezaj, and Javed Ahmed. Deep learning based sequential model for malware analysis using windows exe api calls. PeerJ Computer Science, 6: e285, 2020.
[4] Edward Raff, Jared Sylvester, and Charles Nicholas. Learning the pe header, malware detection with minimal domain knowledge. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 121-132, 2017.
[5] Yong Tang, Bin Xiao, and Xicheng Lu. Signature tree generation for polymorphic worms. IEEE transactions on computers, 60(4):565-579, 2010.
[6] Guangming Liu, Xin Zhou, Jianmin Pang, Feng Yue, Wenfu Liu, and Junchao Wang. Codeformer: A gnn-nested transformer model for binary code similarity detection. *Electronics*, 12 (7):1722, 2023.
[7] Nathaniel Lageman, Eric D Kilmer, Robert J Walls, and Patrick D McDaniel. B in dnn: Resilient function matching using deep learning. In Security and Privacy in Communication Networks: 12th International Conference, pages 517-537. Springer, 2017.
[8] Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao, and Wei Zou.  $\alpha$ diff: cross-version binary code similarity detection with dnn. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 667-678, 2018.
[43] R Vinayakumar, Mamoun Alazab, KP Soman, Prabaharan Poornachandran, and Sitalakshmi Venkatraman. Robust intelligent malware detection using deep learning. IEEE Access, 7: 46717-46738, 2019.
[44] Cangshuai Wu, Jiangyong Shi, Yuexiang Yang, and Wenhua Li. Enhancing machine learning based malware detection model by reinforcement learning. In Proceedings of the 8th International Conference on Communication and Network Security, pages 74-78, 2018.
[45] Huu-Danh Pham, Tuan Dinh Le, and Thanh Nguyen Vu. Static pe malware detection using gradient boosting decision trees algorithm. In Future Data and Security Engineering: 5th International Conference, pages 228-236. Springer, 2018.
[46] Colin Galen and Robert Steele. Evaluating performance maintenance and deterioration over time of machine learning-based malware detection models on the Ember pe dataset. In Seventh International Conference on Social Networks Analysis, Management and Security, pages 1-7. IEEE, 2020.
[47] Robert J. Joyce, Dev Amlani, Charles Nicholas, and Edward Raff. MOTIF: A malware reference dataset with ground truth family labels. Computer Security, 124:102921, 2023.
[48] Silvia Sebastián and Juan Caballero. Avclass2: Massive malware tag extraction from AV labels. In Annual Computer Security Applications Conference, pages 42-53. ACM, 2020.
[49] Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. Avclass: A tool for massive malware labeling. In Fabian Monrose, Marc Dacier, Gregory Blanc, and Joaquín García-Alfaro, editors, Research in Attacks, Intrusions, and Defenses - 19th International Symposium, volume 9854 of Lecture Notes in Computer Science, pages 230-253. Springer, 2016.
[50] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[51] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[52] Fred Damerau. A technique for computer detection and correction of spelling errors. Communications of the ACM, 7(3):171-176, 1964.
[53] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, and Konrad Rieck. DREBIN: effective and explainable detection of android malware in your pocket. In Proceeding of NDSS, 2014.
[54] John E. Moody. Fast learning in multi-resolution hierarchies. In David S. Touretzky, editor, Proceedings of NIPS, pages 29-39, 1988.
[55] Jilei Yang. Fast treeshop: Accelerating SHAP value computation for trees. CoRR, 2021.
A.1 EMBER data set
The original EMBER2017 data set (as reported in the paper introducing it) contains 1.1M samples, with the date of appearance of more than  $95\%$  of them spanning the year 2017. One year later, the authors have collected a second batch of PE files, with  $95\%$  of them appearing in 2018. Our experiments follow the related work and are conducted on the most recent batch of EMBER data (from 2018, with feature version 2, as referenced in the EMBER repository  $^{5}$ ).
Figures 2 and 3 show the sample count and label distribution by date of appearance and subset, respectively. EMBER's authors state that the purpose of the 200K unlabelled samples is to aid further research on unsupervised or semi-supervised learning.
Figure 2: Distribution of sample count per labels with respect to the date of sample first appearance. The vast majority of the samples (95%) appear in 2018.
Figure 3: Distribution of sample count per labels with respect to the data subset.
As a measure of confidence in the label, the work introducing EMBER notes that for each malicious sample, more than 40 vendors available in VirusTotal yield a positive detection result. In Figure 4 we show the correlation between the label and the number of VirusTotal detections. The resulting distribution is in line with our expectations, observing a slight fuzziness in the lower regions of detections, nonetheless respecting the 40-merchant threshold empirically set by the authors. We further note that the unlabelled samples appear to be evenly sampled from each detection bin.
<sup>5</sup>https://github.com/elastic/ember
Figure 4: Distribution of VirusTotal detections per label.
A.2 AVClass2 tags
We turn our attention to metadata obtained from the AVClass2 system  $^{6}$  for extracting, standardizing and ranking tag information from VirusTotal detections. Particularly of interest for our purposes are the malware family (FAM), class (CLASS) and behavior (BEH) tags, given that they can serve as ground truth for evaluating a similarity search method, under the assumption that the metadata should be consistent between a query and its most similar samples. When applied to all 1M samples in the EMBER2018 data set we explored, AVClass2 was able to identify 1219 unique FAM tags, 33 unique CLASS tags, and 43 unique BEH tags. To offer an intuition related to the order of magnitude of the tag frequency, we show the top 20 most prevalent tags by kind in Figures 5, 6, 7.
Figure 5: Top 20 most prevalent FAM tags (log scale).
<https://github.com/malicialab/avclass
Figure 6: Top 20 most prevalent CLASS tags (log scale).
Figure 7: Top 20 most prevalent BEH tags (log scale).
A.3 Evaluation
A.3.1 Label homogeneity
Our first evaluation of the similarity search method proposed in this work concerns the ability of preserving the label between a query and its top  $K$  hits. We regard this evaluation scenario both as a reasonable assumption for Cybersecurity focused similarity search in general and as a sanity check for the method proposed in this paper. We illustrate the results in Figure 8, with histograms showing the label homogeneity for malicious and benign queries, and their top  $K$  hits, with  $K \in \{10, 50, 100\}$ . We observe that the vast majority of the hits are concentrated in the bin close to  $K$ , meaning that almost all returned hits share the same label with the query, which is what we would expect for a good performance.
(a) Top 10.
(b) Top 50.
(c) Top 100.
Figure 8: Number of hits (out of top  $K$ ) matching labels with query.
A.3.2 Relevance @  $K$
Our second evaluation approach implies assessing the relevance between a query and the top  $K$  hits as returned by the proposed similarity search method. The relevance @  $K$  objective evaluates the consistency between two metadata lists. Specifically, we check how well the ranking of FAM tags for a query is preserved in its top  $K$  hits. Results are shown in Figures 9 and 10. We used the following settings: for enriching the tag lists used to construct the tag ranking metadata, we used a common tag co-occurrence threshold of 0.9; in order to penalize inversions in rank, we use the Normalized Edit Similarity function (as described in the main paper). For the histograms, a higher concentration towards 1.0 is better, meaning that almost all  $K$  hits are relevant with respect to the query. For the empirical CDFs, this translates to the curve shifting lower and to the right.
Counterfactual analysis (query = test, knowledge base = train  $\cup$  test): in which we evaluate the effectiveness of the proposed similarity search method knowing the true label of the query samples.
(b) Top 100.
Figure 9: Evaluation results for the counterfactual analysis case.
Unsupervised labelling (query = unlabelled, knowledge base = train): modelling the scenario where a practitioner wants to label unseen samples using the similarity search method. Since the query set does not contain labels, we leverage the presence of malware tags to serve as an indicator instead.
(b) Top 100.
Figure 10: Evaluation results for the unsupervised labelling case.
A.4 Leaf Similarity Extended to other Data Sets
We conduct an additional set of experiments, to address potential generalization concerns that might arise with respect to the leaf similarity approach proposed in this paper. Thus, we use a variant of the DREBIN data set [53]. The DREBIN data set is composed of 129,013 samples, out of which 123,453 are clean and 5,560 malicious.
We perform one-hot encodings on the categorical features in the DREBIN data set, obtaining a total of 550K values. To reduce the amount of computational resources and time required to run this experiment, we apply heuristics (i.e., removing the " * " characters from urls), and then the " hashing trick" [54] to reduce the number of features down to 16,384 values. We perform a 80/20 split of the
data into a train and test subset, while keeping the ratio between clean and dirty samples from the initial DREBIN corpus.
Prior to the final training, we perform hyper-parameter tuning for the XGBoost model, the best results being obtained when the learning rate is 0.05, maximum depth is 12, and number of estimators is equal to 1024. The final XGBoost model obtains a ROC AUC score of 0.95 when evaluated on the test subset. Then we apply the same approach as described in the paper, using leaf similarity class homogeneity check. We obtained the results displayed in Table 6 when considering the "test vs test+train" scenario.
Table 6: Label homogeneity evaluation scenario for leaf similarity over the DREBIN data set.
K	Benign	Malicious	All
Mean	Std	Mean	Std	Mean	Std
1	1	0	1	0	1	0
10	9.96	0.41	8.95	2.30	9.92	0.65
50	49.75	2.19	38.51	16.42	49.27	4.61
100	99.50	4.19	71.21	35.10	98.30	10.10
As it can be observed from these results, our method achieves good performance on another cybersecurity oriented dataset. This discovery makes us confident that as part of our future work we could enrich our current benchmark based on EMBER with other types of malware data.
A.5 Discussion on model interpretability
We briefly address model interpretability for our tree-based model, using a TreeSHAP algorithm [55]. We conduct a quick experiment with the end goal of understanding the connection between the features' importance of a group of samples sharing a class tag and the features' importance for the samples that were enriched with the identical tag. We apply the following steps, and compute:
- the SHAP values for the samples with tag X extracted from AVClass;
- the ranking in terms of importance for the features extracted at the previous step;
- the SHAP values as well as the feature importance ranking for the samples with tag X enriched using our method;
- the Spearman correlation between the group feature ranking and the individual sample feature ranking.
Through random sampling, we obtain a moderate correlation, constantly scoring greater than 0.3 which proves that the sample with generated tags are similar from this perspective to the samples that already attached tags from AVClass. Please note that these results were obtained with a tag co-occurrence threshold of 0.9.
We conclude that a line can be drawn between the enriched tags and the feature importance using SHAP values, but the results are not definitive. Additional research needs to be done in the area of interpretability to better understand the model outputs.
Model interpretability represents an interesting future research topic. From this angle, we perform a very narrow assessment of our similarity method, as outlined in this subsection. However, we believe that this represents a separate line of research that should be the primary focus in a future study.
B.1 EMBERSim data set documentation
The data used in the databank released as part of this research consists of a postprocessing of the EMBER data set, such that we can easily perform binary code similarity search over this data. The original data set consists of feature vectors from 1 million Windows portable executable (PE) samples. The original EMBER release includes a feature extractor which provides 8 groups of diverse features from each of these PE samples.