Automated malware analysis that automatically clusters sandbox reports of similar malware samples

A system and a method for automatically clustering sandbox analysis reports of similar malware samples. An automated malware analysis process includes receiving from a sandbox server the sandbox analysis reports of the similar malware samples at an application programming interface (API) of the clustering server, clustering similar Uniform Resource Locators (URLs) together and clustering the sandbox analysis reports of events in sandbox reports clusters (1-n) based on the URL clustering, static properties of the malware samples and dynamic properties of the malware samples.

BACKGROUND

Aspects of the present invention generally relate to a system and a method for automated malware analysis that automatically clusters sandbox analysis reports of similar malware samples based on soft clustering of Uniform Resource Locators (URLs) to provide a decision support solution that simplifies the manual vetting task of security analysts.

2. Description of the Related Art

A user can report suspicious emails to a cyber security system (several hundred a day). Attachments of reported emails are detonated in a sandbox environment and the generated analysis reports (events), containing URLs, filenames and in general a description of the malware behavior, are used by security analysts to improve the security defenses of an enterprise.

Unfortunately, this process cannot be fully automated, due to the potential presence of false positives samples in the analysis (e.g. wrongly reported email). For this reason, a security analyst must manually vet the output of every sandbox analysis, identify those describing malicious activities and initiating respective mitigation actions (e.g. blacklisting a certain domain at the proxy). This task is performed daily for all the new samples analyzed in the past 24 hours and it is a time-consuming endeavor, requiring the analyst to go through a potentially extensive list of URLs and file indicators to gain insights about the type of activities performed by the analyzed sample. Even the task of determining if an individual URL is malicious or not might be a non-straightforward effort, requiring the analyst to leverage internal and external enrichment resources to determine the right course of action.

On the other hand, different instances of the same malware families tend to exhibit similar behaviors, like connecting to command and control servers with similar names, storing files using similar filenames on the victim host or downloading payload from similar paths on different domains. Indicators generated by similar malware samples usually require the same actions by the analyst and, as such, could be executed in bulk, optimizing the analyst time.

Clustering approaches used in malware analysis has several drawbacks. For example, lexical features extraction method has been used for the clustering problem. A major drawback of using the lexical features is mixed data types. Using a mix of continuous and binary features does not allow using certain distance metrics during classification. Clustering of sandbox analysis reports studied in the literature also has several drawbacks. For example, malware graphs can provide accurate representations but are computationally very expensive to create.

Therefore, there is a need for better automated malware analysis of sandbox analysis reports of malware samples.

SUMMARY

Briefly described, aspects of the present invention relate to a system and a method for automated malware analysis that automatically clusters sandbox analysis reports of similar malware samples based on soft clustering of Uniform Resource Locators (URLs) to provide a decision support solution that simplifies the manual vetting task of security analysts. A malware database has been utilized as a main source of data for the present invention. Database has been extracted in JavaScript Object Notation (JSON) format and sandbox reports which include URL connections has been identified. In computing, JavaScript Object Notation (JSON) is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute-value pairs and array data types (or any other serializable value). As a first of the clustering of URLs, a text encodings-based feature extraction method has been used for the clustering problem by an automated malware analysis system. The automated malware analysis system uses an auto encoder-based approach to obtain a reduced dimension, continuous representation of the mixed data type feature space. This approach supports the use of different clustering algorithms and a richer set of distance metrics. Using the feature extraction method described above, the automated malware analysis system obtains features to be used in clustering of the URLs. The automated malware analysis system can use different clustering approaches such as Density based spatial clustering of applications with noise (DBSCAN) (distance based), k-means and Gaussian mixture models (GMMs). Due to uncertain nature of the problem, GMM is used as the best option as it allows probabilistic clustering of the URLs. At the next stage, the automated malware analysis system uses the clustering of URLs as features for clustering the sandbox reports: together with average clustering probabilities of URLs, other event features such as timestamp of the analysis, static properties of the sample and dynamic properties. Due to mixed data type of event features, the automated malware analysis system uses Gower's distance for calculating a distance matrix. Final clustering of events is determined using density based spatial clustering.

In accordance with one illustrative embodiment of the present invention, an automated malware analysis system is provided for automatically clustering sandbox analysis reports of similar malware samples. The system comprises a sandbox server including a first processor and a first memory. The first memory stores a sandbox virtual machine comprising software instructions. These software instructions, when executed by the first processor, generate the sandbox analysis reports of the similar malware samples. The system further comprises a clustering server including a second processor and a second memory. The second memory stores a Uniform Resource Locator (URL) categorizer comprising software instructions. These software instructions, when executed by the second processor, cluster similar URLs together. The second memory further stores a report categorizer comprising software instructions. These software instructions, when executed by the second processor, cluster the sandbox analysis reports of events based on the URL clustering performed by the URL categorizer, static properties of the malware samples and dynamic properties of the malware samples.

In accordance with another illustrative embodiment of the present invention, a non-transitory computer readable medium comprising computer instructions capable of being executed in a processor of a clustering server of an automated malware analysis system is provided. The computer instructions configured to receive from a sandbox server sandbox analysis reports of similar malware samples at an application programming interface (API) of the clustering server for automatically clustering the sandbox analysis reports of the similar malware samples. The computer instructions further configured to cluster similar Uniform Resource Locators (URLs) together. The computer instructions further configured to cluster the sandbox analysis reports of events based on the URL clustering, static properties of the malware samples and dynamic properties of the malware samples. The computer instructions further configured to send all sandbox reports clusters (1-n) from the clustering server to the sandbox server so that an analyst can focus only on one event per cluster and quickly verify correctness of a same decision for all similar events in a same cluster.

DETAILED DESCRIPTION

To facilitate an understanding of embodiments, principles, and features of the present invention, they are explained hereinafter with reference to implementation in illustrative embodiments. In particular, they are described in the context of a system and a method for automated malware analysis that automatically clusters sandbox analysis reports of similar malware samples based on soft clustering of Uniform Resource Locators (URLs) to provide a decision support solution that simplifies the manual vetting task of security analysts. An automated malware analysis system comprises a sandbox server including a first processor and a first memory and a clustering server including a second processor and a second memory. The sandbox server includes a sandbox virtual machine to generate the sandbox reports of the similar malware samples. The clustering server includes a Uniform Resource Locator (URL) categorizer to cluster similar URLs together. The clustering server further includes a report categorizer to cluster analysis reports of events based on the URL clustering performed by the URL categorizer, static properties of the malware samples and dynamic properties of the malware samples. In this way, a decision support solution is provided for security analysts to simplify the manual vetting task by automatically clustering sandbox reports of similar malware samples. Using this automated malware analysis process the analyst can focus only on one event per cluster and quickly verify the correctness of the same decision for all the similar events in the same cluster. At a high level, the decision support solution consists of a 2-step approach: step 1: similar URLs are clustered together (e.g. URLs generated by similar domain generation algorithms (DGAs), known legitimate websites like microsoft.com, etc.) using an auto encoder based feature extraction as input to GMMs; and step 2: analysis reports are clustered based on three main set of features: the URL clustering performed at step 1, static properties of the sample (e.g. Yet Another Recursive/Ridiculous Acronym (YARA)-rules matches) and other dynamic properties of the malware (e.g. disk access during execution). YARA rules are a way of identifying malware (or other files) by creating rules that look for certain characteristics. With YARA descriptions of malware families can be created based on textual or binary patterns. YARA is a tool used to identify and classify malware samples. YARA identifies and classifies malware based on custom rules created in a platform. A rule is a description based on textual or binary patterns. Embodiments of the present invention, however, are not limited to use in the described devices or methods.

These and other embodiments of an automated malware analysis system according to the present disclosure are described below with reference toFIGS.1-7herein. Like reference numerals used in the drawings identify similar or identical elements throughout the several views. The drawings are not necessarily drawn to scale.

Consistent with one embodiment of the present invention,FIG.1represents a block diagram of an automated malware analysis system105that provides automatic clustering of sandbox analysis reports of similar malware samples based on soft clustering of Uniform Resource Locators (URLs) in accordance with an exemplary embodiment of the present invention. The automated malware analysis system105is configured to automatically cluster sandbox analysis reports107(1-n) of similar malware samples110(1-n) based on soft clustering of Uniform Resource Locators (URLs)112to provide a decision support solution that simplifies the manual vetting task of security analysts. For being able to analyze a malware sample110, the automated malware analysis system105has a runtime environment in place. A file or an email attachment infected by a virus is an example of the malware sample110.

The automated malware analysis system105performs a sandboxing analysis to generate sandbox analysis reports107(1-n) in JavaScript object notation (JSON) format. A Sandbox Analysis Report107provides insight as to what occurred during a sandbox process and any signatures that lead to its scan result. For example, a Sandbox Analysis Report package may provide detection results for known and new advanced threats. It includes static and multiple dynamic analyses of uploaded files in an array of sandboxes. In addition, files that were uploaded and were sent for a sandbox analysis will result with a sandbox report. A sandbox is a testing environment that executes potentially malicious files or URL requests in an isolated area, typically on a virtual machine. If the sandboxing application finds that an executed file modified system files or infected the system in any way, those issues will not spread to other areas. Files are executed in their own sequestered area, where they can be tested without posing any threat to a client computer or network. Because the environment is not actually connected to a network, any malware that executes in the sandbox environment cannot infect a real device or network. The automated malware analysis system105evaluates the threat of a given file in one or more Windows virtual machines or emulated virtual machines and provides a reputation score as a number between 1 and 100. The higher the number, the greater the threat. Sandboxing may be a cloud-based service that uses static code analysis and behavioral analysis to detect advanced threats. Sandbox services use different methods to identify the actions an executable file would take on a client workstation, including malicious URL web requests and changes to system files. Once a file is analyzed, sandbox services score the file and report it.

In the automated malware analysis system105, sandbox gets a file, executes it automatically, and disables warning and basic security features in order to allow the malware to exhibit its malicious behavior. Then records network connection and other actions performed by the malware. This can be done fully automated or interactively. The latter approach is used in some analysis where the malware requires some form of user interaction in order to detonate. Interaction is achieved by having a user clicking on buttons/popups/windows in a virtual machine.

The automated malware analysis system105comprises a sandbox server115(1) including a first processor117(1) and a first memory120(1) and a clustering server115(2) including a second processor117(2) and a second memory120(2). The sandbox server115(1) is a server machine hosting a sandbox virtual machine122where binary samples are executed. Each execution generates a new sandbox analysis report,107which is sent via an API call to the clustering server115(2). The clustering server115(2) is a server machine hosting a URL categorizer125software and a report categorizer127software. Once the clustering server115(2) receives the sandbox analysis reports107(1-n) from the sandbox server115(1) via an API, it processes them using the two categorizers and it returns the new report clustering to the sandbox server115(1) via an API call. Application Programming Interface (API) is an interface through which the server functionalities are triggered.

The sandbox server115(1) includes the sandbox virtual machine122to generate the sandbox analysis reports107(1-n) of the similar malware samples110(1-n). The clustering server115(2) includes the Uniform Resource Locator (URL) categorizer125to cluster similar URLs112together. The clustering server115(2) further includes the report categorizer127to cluster the sandbox analysis reports107(1-n) of events based on the URL clustering performed by the URL categorizer125, static properties130(1) of the malware samples110(1-n) and dynamic properties130(2) of the malware samples110(1-n). In this description, the term Event is used as the technical representation of a sandbox analysis report107. In simple terms, event is another word for “sandbox analysis report”.

The first memory120(1) stores the sandbox virtual machine122which comprises software instructions when executed by the first processor117(1) it generates the sandbox analysis reports107(1-n) of the similar malware samples110(1-n). The second memory120(2) stores the Uniform Resource Locator (URL) categorizer125which comprises software instructions when executed by the second processor117(2) it clusters similar URLs112together. The second memory120(2) further stores the report categorizer127which comprises software instructions when executed by the second processor117(2) it clusters the sandbox analysis reports107(1-n) of events based on the URL clustering performed by the URL categorizer125, the static properties130(1) of the malware samples and the dynamic properties130(2) of the malware samples110(1-n). Other static and dynamic properties or features130(1-2) include any other information in the sandbox analysis report107that is not a URL.

The sandbox virtual machine122is a virtual host, running standard technologies, that emulates via software the behavior of a physical machine. A sandbox is a virtual environment where the malware samples110(1-n) can be safely “detonated”, i.e. executed to trigger the malicious behavior, without effects on the host system. A sandbox analysis report107is a detailed report about the execution of a binary sample within the sandbox. A sandbox analysis report107includes static information about the sample (like file size, filename, compile time, etc. . . . ) as well as details about its execution (e.g. the sequence of system calls invoked over time, the list of resources allocated, files read and written, etc.) A URL112is a location or address identifying where documents can be found on the Internet. In the context of the present invention, the URLs112are extracted from a sandbox analysis report107and they indicate the remote resources, like files or webpages, that a binary sample attempted accessing during its execution.

The URL categorizer125is a component that categorizes a list of URLs using advanced machine learning techniques. The URL categorizer125receives in input a list of URLs and return a list of URL lists, divided by categories according to URL similarity. The definition of category and similarity may be specified based on the description of the various internal components.

The sandbox server115(1) further comprises a first application programming interface (API)132(1) to receive a sandbox reports cluster1135(1), a sandbox reports cluster2135(2), a sandbox reports cluster3135(3) and . . . a sandbox reports cluster n135(n) from the clustering server115(2). The clustering server115(2) further comprises a second application programming interface (API)132(2) to receive the sandbox analysis reports107(1-n) from the sandbox server115(1).

The URL categorizer125to soft cluster the similar URLs112together using an auto-encoder network-based feature extraction as input to a Gaussian mixture model (GMM). Auto-encoder network can be used for dimensionality reduction of data and for re-construction of sample population with unknown, complex multivariate probability distribution, where small-probability samples have little contribution to the auto-encoder network, leading to high re-construction error. For example, a deep “auto-encoder” network structure may be trained to learn low-dimensional codes from high-dimensional input vectors. A Gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. Gaussian mixture models are a probabilistic model for representing normally distributed subpopulations within an overall population. Mixture models in general don't require knowing which subpopulation a data point belongs to, allowing the model to learn the subpopulations automatically. A GMM attempts to find a mixture of multi-dimensional Gaussian probability distributions that best model any input dataset. The mixture of Gaussians, a.k.a. GMM is used for density estimation and clustering.

GMM is a clustering method based upon linear learning models. In particular, given a set of training data XL xM, where L is the dimension of the data and M is the number of samples, the clustering method learns K centroids such that each sample can be assigned to the closest centroid. Suppose the observed feature vectors form a feature space and the appropriate K centroids in the high-dimensional feature space are known. A typical pipeline defines a function f: RL→RK that maps the observed L-dimensional feature vector to a K-dimensional feature vector (K<L). For instance, first calculate the affiliations for each observed feature vector (w.r.t. the K centroids) and then use such affiliations as morphological signatures to represent each key point in the feature space. Given a Gaussian mixture model, the goal is to maximize the likelihood function w.r.t. the parameters (means, covariances and mixing coefficients). Clustering with a GMM may be done using the Statistics and Machine Learning and specifying optional parameters when fitting the GMM model.

GMMs are used for data clustering. The automated malware analysis system105can use GMMs to perform either hard clustering or soft clustering on query data. To perform hard clustering, the GMM assigns query data points to the multivariate normal components that maximize the component posterior probability, given the data. That is, given a fitted GMM, the automated malware analysis system105assigns query data to the component yielding the highest posterior probability. Hard clustering assigns a data point to exactly one cluster. Additionally, the automated malware analysis system105can preferably use a GMM to perform a more flexible clustering on data, referred to as soft (or fuzzy) clustering. Soft clustering methods assign a score to a data point for each cluster. The value of the score indicates the association strength of the data point to the cluster. As opposed to hard clustering methods, soft clustering methods are flexible because they can assign a data point to more than one cluster. When the automated malware analysis system105performs GMM clustering, the score is the posterior probability. GMM clustering can accommodate clusters that have different sizes and correlation structures within them. Therefore, GMM clustering can be more appropriate than methods such as k-means clustering. Like many clustering methods, GMM clustering requires one to specify the number of clusters before fitting the model. The number of clusters specifies the number of components in the GMM.

The report categorizer127is configured to cluster the sandbox analysis reports107(1-n) of events so that an analyst can focus only on one event per cluster and quickly verify correctness of a same decision for all similar events in a same cluster. The report categorizer127is a component that given in input a list of sandbox analysis reports and a categorization of URLs returns a partition of the sandbox analysis reports. Each partition (cluster) represents a different category of sandbox analysis reports. Sandbox report clusters imply a group of sandbox analysis reports belonging to the same category, according to the report categorizer127. In this context, a category is a set of sandbox analysis reports107with similar properties that usually identify samples of malware belonging to the same phishing campaigns.

The URL categorizer125to soft cluster the similar URLs by performing text encoding. The text encoding uses a standard for the representation of texts in digital form. For example, an encoding standard is a numbering scheme that assigns each text character in a character set to a numeric value. A character set can include alphabetical characters, numbers, and other symbols. Encoding is the process of converting data from one form to another. Whenever data is encoded, it can only be read by a program that supports the correct type of encoding. Most text editors support multiple types of text encoding, so it is rare to find a text file that will not open in a standard text editor. However, if a text editor does not support the encoding used in a text document, some or all of the characters may appear as strange symbols rather than the intended text.

The URL categorizer125to perform feature extraction after the text encoding. The URL categorizer125to perform a GMM based soft clustering of the similar URLs112after the feature extraction.

The report categorizer127to determine an event feature matrix from clustered URLs140. The clustered URLs140may be a list of URLs grouped by a category. In this context, a category is a set of sandbox analysis reports107with similar properties that usually identify samples of malware belonging to the same phishing campaigns.

The report categorizer127to determine a Gower's distance matrix from the event feature matrix. The first step involves calculating the Gower's distance matrix to separate sandbox analysis reports107into (dis)similar groups from event feature data. For example, a Gower's distance matrix may be generated for the uncorrelated event features, with distance measure ranging from, e.g., 0 to 1. With the Gower's distance matrix, one can visually compare event features between sandbox analysis reports107. Results are calculated by using a Gower's distance matrix derived from sandbox analysis reports107(dis)similarities based on properties that usually identify samples of malware belonging to the same or different phishing campaigns. Gower's distance matrix is a non-estimation method and the Gower's distance matrix can be used to detect detectable differences as the Gower's distance matrix can identify differences of each sandbox analysis report in a multidimensional functional space. Data sets may be analyzed using the non-estimation method (Gower's distance matrix) by calculating the Gower's distance matrix among observations to identify the diverse pairs of sandbox analysis reports107. Gower's distance matrix is used for similarity calculations as it has shown to be more reliable for mixed data with a preponderance of weighted binary data. Cluster analysis may be used to examine overall patterning within a Gower's distance matrix. Clusters of sandbox analysis reports107may be predicted to fall into distinct spatial groups.

The report categorizer127to perform a density-based spatial clustering after determining the Gower's distance matrix using a Density-based spatial clustering of applications with noise (DBSCAN) algorithm. DBSCAN is a data clustering algorithm that is used for data mining and machine learning. Based on a set of points (e.g., in a bidimensional space), DBSCAN groups together points that are close to each other based on a distance measurement and a minimum number of points. It also marks as outliers the points that are in low-density regions.

The DBSCAN algorithm basically requires 2 parameters:

eps: specifies how close points should be to each other to be considered a part of a cluster. It means that if the distance between two points is lower or equal to this value (eps), these points are considered neighbors.

minPoints: the minimum number of points to form a dense region. For example, if one sets the minPoints parameter as 5, then one needs at least 5 points to form a dense region.

The DBSCAN algorithm is used to find associations and structures in data that are hard to find manually but that can be relevant and useful to find patterns and predict trends such as in clustering. It is a density-based clustering non-parametric algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away).

Consider a set of points in some space to be clustered. Let c be a parameter specifying the radius of a neighborhood with respect to some point. For the purpose of DBSCAN clustering, the points are classified as core points, (density-)reachable points and outliers, as follows:

A point p is a core point if at least minPts points are within distance c of it (including p).

A point q is directly reachable from p if point q is within distance c from core point p. Points are only said to be directly reachable from core points.

A point q is reachable from p if there is a path p1, . . . , pnwith p1=p and pn=q, where each pi+1is directly reachable from pi. Note that this implies that all points on the path must be core points, with the possible exception of q.

All points not reachable from any other point are outliers or noise points.

Now if p is a core point, then it forms a cluster together with all points (core or non-core) that are reachable from it. Each cluster contains at least one core point; non-core points can be part of a cluster, but they form its “edge”, since they cannot be used to reach more points.

Reachability is not a symmetric relation since, by definition, no point may be reachable from a non-core point, regardless of distance (so a non-core point may be reachable, but nothing can be reached from it). Therefore, a further notion of connectedness is needed to formally define the extent of the clusters found by DBSCAN.

Two points p and q are density-connected if there is a point o such that both p and q are reachable from o. Density-connectedness is symmetric. A cluster then satisfies two properties: all points within the cluster are mutually density-connected and if a point is density-reachable from any point of the cluster, it is part of the cluster as well.

The report categorizer127to output the sandbox reports clusters1-n135(1-n). Clustering, an unsupervised machine learning technique that aims to grouping analogous entities into one cluster and partitioning the dissimilar objects into another cluster. A cluster is defined as a subset of similar objects, defined by certain parameters, within a larger set.

Referring toFIG.2, it illustrates a block diagram of an automated malware analysis process205in accordance with an exemplary embodiment of the present invention. A sandbox virtual machine207generates a plurality of sandbox analysis reports210(1-m) of similar malware samples. The plurality of sandbox analysis reports210(1-m) contain URLs212which are sent to a URL categorizer215. The URL categorizer215performs text encoding in step218(1), performs feature extraction in step218(2) and performs GMM-based URL's soft clustering in step218(3) to generate clustered URLs220. The clustered URLs220and other static and dynamic features225are sent to a report categorizer227to generate the sandbox reports cluster1-n135(1-n). The report categorizer227calculates an event feature matrix in step230(1), calculates a Gower's distance matrix in step230(2) and performs density-based spatial clustering in step230(3).

There are two key steps of the present approach: 1. auto-encoder network-based feature extraction used as input to GMMS for soft clustering of URLs. and 2. using the soft clustering for a higher-level clustering of related events. In particular,FIG.3shows a pipeline of the proposed solution. Given a URL, the automated malware analysis process205tokenizes the URL using a 100-character dictionary. The tokenization is used as input to an auto-encoder network. At an embedding layer of the auto-encoder network, each character is embedded to a n-dimensional vector which is denoted as “embedding dimension. This embedding is initialized randomly and learned through a training phase. The automated malware analysis process205uses a bidirectional LSTM model due to its success in capturing sequences. the automated malware analysis process205trains the auto-encoder network with a batch size of 32 in 50 epochs. As the final model, the automated malware analysis process205chooses the model with the best validation accuracy. The automated malware analysis process205extracts the intermediate features at a LSTM layer for clustering with GMMs. The automated malware analysis process205uses AIC and BIC scores to determine the optimal number of clusters.

At the next stage, the automated malware analysis process205uses the clustering of URLs112as features for clustering the events: together with average clustering probabilities of URLs, the automated malware analysis process205uses other event features such as timestamp of the analysis (similar malware tend to be delivered within the same phishing campaign), the static properties130(1) of the sample (e.g. filename, format, whether it's encrypted or not, one hot encoding of YARA rules matches, file size) and the dynamic properties130(2) (e.g. number of dropped files, number of URLs per event, accesses performed). Due to mixed data type of event features, the automated malware analysis process205uses Gower's distance for calculating the distance matrix. Final clustering of events is determined using density based spatial clustering.

Turning now toFIG.3, which illustrates a pipeline305of Uniform Resource Locator (URL) clustering in accordance with an exemplary embodiment of the present invention. The pipeline305of Uniform Resource Locator (URL) clustering includes a step310(1) of character level encoding, a step310(2) of using an auto-encoder network, a step310(3) of extracting a features matrix and a step310(4) of GMM-based soft clustering. The automated malware analysis system105ofFIG.1can preferably use a GMM to perform a more flexible clustering on data, referred to as soft (or fuzzy) clustering. Soft clustering methods assign a score to a data point for each cluster. The value of the score indicates the association strength of the data point to the cluster. As opposed to hard clustering methods, soft clustering methods are flexible because they can assign a data point to more than one cluster. When the automated malware analysis system105performs GMM clustering, the score is the posterior probability. GMM clustering can accommodate clusters that have different sizes and correlation structures within them. Therefore, GMM clustering can be more appropriate than methods such as k-means clustering. Like many clustering methods, GMM clustering requires one to specify the number of clusters before fitting the model. The number of clusters specifies the number of components in the GMM.

FIG.4illustrates soft clustering of URLs using GMMs in accordance with an exemplary embodiment of the present invention. InFIG.4, a “website” information is redacted, i.e. www.website.com becomes www.<redacted URL1>.com. Clustering, an unsupervised machine learning technique that aims to grouping analogous entities into one cluster and partitioning the dissimilar objects into another cluster. A cluster is defined as a subset of similar objects, defined by certain parameters, within a larger set. Gaussian mixture model (GMM) is a clustering method based upon linear learning models. In particular, given a set of training data XL×M, where L is the dimension of the data and M is the number of samples, the clustering method learns K centroids such that each sample can be assigned to the closest centroid. Suppose the observed feature vectors form a feature space and the appropriate K centroids in the high-dimensional feature space are known. For example, a pipeline defines a function f: RL→RK that maps the observed L-dimensional feature vector to a K-dimensional feature vector (K<L).

The automated malware analysis system105can preferably use a GMM to perform a more flexible clustering on data, referred to as soft clustering. Soft clustering methods assign a score to a data point for each cluster. The value of the score indicates the association strength of the data point to the cluster. Soft clustering methods are flexible because they can assign a data point to more than one cluster. When the automated malware analysis system105performs GMM clustering, the score is the posterior probability. GMM clustering can accommodate clusters that have different sizes and correlation structures within them.

As seen inFIG.5, a pipeline505of event clustering for clustering sandbox analysis reports107is illustrated in accordance with an exemplary embodiment of the present invention. The pipeline505of event clustering for clustering sandbox analysis reports107includes a step510(1) of calculating an event features matrix from the GMM-based soft clustering of URLs, a step510(2) of calculating a distance matrix based on Gower's distance and a step510(3) of performing DBSCAN clustering based on density.

FIG.6illustrates a schematic view of a flow chart of a method600for automatically clustering the sandbox analysis reports107of the similar malware samples110in the automated malware analysis system105in accordance with an exemplary embodiment of the present invention. Reference is made to the elements and features described inFIGS.1-5. It should be appreciated that some steps are not required to be performed in any particular order, and that some steps are optional.

The method600comprises a step605of receiving from the sandbox server115(1) the sandbox analysis reports107of the similar malware samples110at an application programming interface (API) of the clustering server115(2) for automatically clustering the sandbox analysis reports107of the similar malware samples110. The method600further comprises a step610of clustering similar Uniform Resource Locators (URLs)112together. The method600further comprises a step615of clustering the sandbox analysis reports107of events based on the URL clustering, the static properties130(1) of the malware samples110and the dynamic properties130(2) of the malware samples110. The method600further comprises a step620of sending the sandbox reports clusters1-n135(1-n) from the clustering server115(2) to the sandbox server115(1) so that an analyst can focus only on one event per cluster and quickly verify correctness of a same decision for all similar events in a same cluster.

There are multiple advantages of the solution at URL level clustering and event level clustering. At the URL level, use of auto-encoders enables a compact accurate representation of the URL in the continuous space. This continuous representation enables use of different distance metrics (e.g., Euclidean together with k-means). Auto-encoder based feature representation together with Gaussian mixture models, achieves a soft clustering of the URLs. A URL might be similar to multiple other class of URLs due to certain features like, page extension, entropy, length etc. With soft clustering, the confidence of the algorithm in assigning a URL in certain clusters. At the event level clustering, a multilevel clustering for malwares is used. This method supports mixed data clustering. Advantages resulting from the present invention can serve as a decision aid to security analysts to perform URL and event analysis in a faster and more informed way. Accordingly, a decision support solution is provided for security analysts to simplify the manual vetting task by automatically clustering the sandbox analysis reports107of the similar malware samples110. The analyst can focus only on one event per cluster and quickly verify the correctness of the same decision for all the similar events in the same cluster.

FIG.7shows an example of a computing environment700within which embodiments of the disclosure may be implemented. The computing environment700includes a computer system710that may include a communication mechanism such as a system bus721or other communication mechanism for communicating information within the computer system710. The computer system710further includes one or more processors720coupled with the system bus721for processing the information.

The processors720may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as described herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may include any type of suitable processing unit including, but not limited to, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s)720may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor may be capable of supporting any of a variety of instruction sets. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

Continuing with reference toFIG.7, the computer system710may also include a system memory730coupled to the system bus721for storing information and instructions to be executed by processors720. The system memory730may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM)731and/or random access memory (RAM)732. The RAM732may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM731may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory730may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors720. A basic input/output system733(BIOS) containing the basic routines that help to transfer information between elements within computer system710, such as during start-up, may be stored in the ROM731. RAM732may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors720. System memory730may additionally include, for example, operating system734, application programs735, and other program modules736. Application programs735may also include a user portal for development of the application program, allowing input parameters to be entered and modified as necessary.

The operating system734may be loaded into the memory730and may provide an interface between other application software executing on the computer system710and hardware resources of the computer system710. More specifically, the operating system734may include a set of computer-executable instructions for managing hardware resources of the computer system710and for providing common services to other application programs (e.g., managing memory allocation among various application programs). In certain example embodiments, the operating system734may control execution of one or more of the program modules depicted as being stored in the data storage740. The operating system734may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

The computer system710may also include a disk/media controller743coupled to the system bus721to control one or more storage devices for storing information and instructions, such as a magnetic hard disk741and/or a removable media drive742(e.g., floppy disk drive, compact disc drive, tape drive, flash drive, and/or solid state drive). Storage devices740may be added to the computer system710using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire). Storage devices741,742may be external to the computer system710.

The computer system710may also include a field device interface765coupled to the system bus721to control a field device766, such as a device used in a production line. The computer system710may include a user input interface or GUI761, which may comprise one or more input devices, such as a keyboard, touchscreen, tablet and/or a pointing device, for interacting with a computer user and providing information to the processors720.

The computer system710may perform a portion or all of the processing steps of embodiments of the invention in response to the processors720executing one or more sequences of one or more instructions contained in a memory, such as the system memory730. Such instructions may be read into the system memory730from another computer readable medium of storage740, such as the magnetic hard disk741or the removable media drive742. The magnetic hard disk741and/or removable media drive742may contain one or more data stores and data files used by embodiments of the present disclosure. The data store740may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores in which data is stored on more than one node of a computer network, peer-to-peer network data stores, or the like. The data stores may store various types of data such as, for example, skill data, sensor data, or any other data generated in accordance with the embodiments of the disclosure. Data store contents and data files may be encrypted to improve security. The processors720may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory730. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

The computing environment700may further include the computer system710operating in a networked environment using logical connections to one or more remote computers, such as remote computing device780. The network interface770may enable communication, for example, with other remote devices780or systems and/or the storage devices741,742via the network771. Remote computing device780may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system710. When used in a networking environment, computer system710may include modem772for establishing communications over a network771, such as the Internet. Modem772may be connected to system bus721via user network interface770, or via another appropriate mechanism.

Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and/or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”

While a servers-based architecture of an automated malware analysis system is described here a range of one or more other types of automated malware analysis systems or other forms of automated malware analysis systems are also contemplated by the present invention. For example, other types of automated malware analysis systems may be implemented based on one or more features presented above without deviating from the spirit of the present invention.

The techniques described herein can be particularly useful for automatically clustering sandbox analysis reports of similar malware samples based on GMM-based soft clustering of URLs. While particular embodiments are described in terms of the soft clustering of URLs, the techniques described herein are not limited to soft clustering of URLs but can also be used with other methods of clustering of URLs.

Additionally, any examples or illustrations given herein are not to be regarded in any way as restrictions on, limits to, or express definitions of, any term or terms with which they are utilized. Instead, these examples or illustrations are to be regarded as being described with respect to one particular embodiment and as illustrative only. Those of ordinary skill in the art will appreciate that any term or terms with which these examples or illustrations are utilized will encompass other embodiments which may or may not be given therewith or elsewhere in the specification and all such embodiments are intended to be included within the scope of that term or terms.

Respective appearances of the phrases “in one embodiment,” “in an embodiment,” or “in a specific embodiment” or similar terminology in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any particular embodiment may be combined in any suitable manner with one or more other embodiments. It is to be understood that other variations and modifications of the embodiments described and illustrated herein are possible in light of the teachings herein and are to be considered as part of the spirit and scope of the invention.

It will also be appreciated that one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any component(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component.