SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the original formulation, the authors included malicious and benign labels, perfectly balanced among classes along with family tags derived from the first version on the AVClass package. In the most up to date version of the data set only 800K out of 1 million samples are tagged as being either malicious or benign respectively. We use all the conventional features from the original implementation as well as the benign and malicious labels. As part of our contribution, we enhance each sample in EMBER with similarity-informed tags extracted using both an up to date second iteration of the AVClass package (for which we get tags of kinds FAM, CLASS, BEH, FILE) as well as the top 100 most similar samples to a given query according to our leaf similarity method. We provide an extensive evaluation of the similarity method proposed in this paper, showing that it ensures both label and family tag consistency. Thus, this research results in 1 million fully tagged with malware families and labeled (i.e. benign/malicious) samples along with 400k samples for which the top 100 most similar samples are included. Please note that the most similar samples can also be deduced for the rest of the 600K samples based on the information provided. However, our focus is to keep these samples as an indexed database of known samples among which we can search for similar samples.
We propose a split designed to simulate production use cases as closely as possible. We first construct a train split (identical to the EMBER labelled training corpus) on which we propose training similarity informed models. Then we break our queries into two different splits. The first split supports a validation scenario in which we query an indexed database with all the samples seen during training. In this case both the queries and the indexed database span the same time intervals. The second split supports a production-like evaluation scenario, in which we search through an indexed database using queries collected in a time interval disjoint with the interval of the training subset. The splits corresponding to the aforementioned two scenarios, are published using the CSV format.
B.2 Intended uses
We hope that our proposed contribution will encourage further research in the area of binary code similarity given the large scale corpus we have put together for this purpose. Our intention is to release both our similarity informed tagging technique as well as our similarity method such that upcoming research can construct novel similarity algorithms for binary code using a realistic setting.
B.3 Data set download
Instructions to download all necessary artifacts as well as reproduce our experiments are provided at: https://github.com/CrowdStrike/embersim-databank. This public GitHub repository links to all the other required open-source resources used in this work and also contains all the code necessary to transform the original data into our format as well as all the sample code required to evaluate our method following the details presented in our paper.
B.4 Author responsibility for violation of rights
Our proposed contribution does not contain any kind of sensitive data, personally identifiable information or confidential information. The authors are not aware of any possible violation of rights and take responsibility for the published data.
B.5 Data set hosting and long-term preservation
The authors take full responsibility for the availability of the processed data in the provided repository. No statement can be made about the availability of third-party dependencies such as the EMBER data set or the AVClass repository. However, all the samples (binary files) part of the EMBER data set are
available for download using VirusTotal. VirusTotal is a service free of charge for non-commercial use and is supported by both private companies as well as public contributors as a central source of truth for the cybersecurity community. We make sure to include all the tag information for these samples in our repository which is stored on Zenodo and will continue to be available as long as CERN exists.
B.6 License
We release our code under a AGPL V3 Clause License, therefore allowing the redistribution and use in source and binary forms, with or without modification as long as the resulting code remains subject to the same license. The metadata is released under a Creative Commons Attribution 4.0 International Public License allowing others to freely share, copy and redistribute the material in any medium or format.