Automatic generation of generic file signatures

Systems and methods to automatically generate signatures used to detect malware are provided. The systems and methods use machine learning techniques, to build an over-trained heuristic model to analyze software, cluster identified patterns, validate the clusters against known reputational metrics, automatically create signatures and, in some examples, deploy such signatures to remote computing devices.

BACKGROUND

Field

The present disclosure generally relates to computer security, and in particular to generating generic file signatures for detecting malicious software.

Description of the Related Art

Malicious software, sometimes called “malware,” is generally defined as software that executes on a computing system surreptitiously, or that has some surreptitious functionality. Malware can take many forms, such as parasitic viruses that attach to legitimate files, worms that exploit weaknesses in the computer's security in order to infect the computer and spread to other computers, Trojan horse programs that appear legitimate, but actually contain hidden malicious code, and spyware that monitors keystrokes and/or other actions on the computer in order to capture sensitive information or display advertisements. A wide variety of malicious software (malware) can attack modem computers. Malicious entities sometimes attack servers that store sensitive or confidential data that can be used to the malicious entity's own advantage. Similarly, other computers, including home computers, must be constantly protected from malicious software that can be transmitted when a user communicates with others via electronic mail, when a user downloads new programs or program updates, and in many other situations.

Conventional techniques for detecting malware, such as signature string scanning, are part of an overall computer security protection regime, but less effective against today's malware. Modern malware is often targeted and delivered to only a relative handful of computers. For example, a Trojan horse program can be designed to target computers in a particular department of a particular enterprise. Such malware might never be encountered by security analysts, and thus the security software might never be configured with signatures for detecting such malware. Mass-distributed malware, in turn, can contain polymorphisms that make every instance of the malicious software unique. As a result, it is difficult to develop signature strings that reliably detect all instances of the malware.

Newer techniques for detecting malware apply rules that make an inference about whether a target computer file is malicious by examining dynamic attributes of the target file, code or software. This type of malware detection uses a set of heuristics to make the inference based off dynamic file attributes and then generate signatures (sometimes called behavioral signatures) to identify malware. It should be noted that the terms “heuristic” or “heuristic algorithm” as used herein, generally refer to any type or form of algorithm, formula, model, or tool that may be used to classify or make decisions with respect to an object or sample.

The signatures are typically derived from decision trees developed using decision tree induction algorithms. Decision trees and other heuristics may be trained and refined using a corpus of known samples. As an example for detecting malware, a security-software vendor may train a malware detection heuristic by applying the heuristic to a corpus of samples containing known-malicious files and known-legitimate files. Known-legitimate files refer to software known to be non-malicious, and are sometimes referred to as “goodware.” Goodware can include common and/or popular software programs that are frequently present on a computer system.

The accuracy of a heuristic is often limited by the size of the corpus of samples used to train the heuristic. As such, heuristics may generate false negatives and/or false positives upon being deployed and used in the real world. The term “false positive” may represent an error made in rejecting a null hypothesis when the null hypothesis is actually true. For example, a malware-detection heuristic may produce a false positive by incorrectly determining that a legitimate file or software application is malicious. In order to improve the accuracy of a heuristic, heuristic providers typically: 1) add misclassified samples gathered from the field to the corpus of samples used to train the heuristic, 2) re-train the heuristic using the modified corpus of samples, and then 3) redeploy the re-trained heuristic. However, even if a heuristic is re-trained using a corpus of samples that includes misclassified samples gathered from the field, re-trained heuristics may produce new false positives upon being redeployed in the field. Because of this, heuristic providers may have to constantly redeploy and retest a heuristic until satisfactory performance is obtained.

SUMMARY

The present disclosure provides a method, computer system, and computer-readable storage medium for generating signatures for detecting malware. In one embodiment, the method includes collecting a set of static attributes from a malware dataset and a goodware dataset, and generating one or more trained or over-trained decision trees from the set of attributes. The amount of overtraining effects the overall specificity and accuracy of the resulting signatures. Once the one or more decision trees are generated, data from a collection of known goodware and unknown files is processed through the one or more decision trees to identify patterns and form one or more clusters. Each of the one or more clusters is validated against a reputation value range to determine if each of the one or more clusters is a bad cluster or a suspected bad cluster, and generating a signature associated with each bad cluster and each suspected bad cluster. The method may also include a step of deploying each signature to one or more remote computing systems.

The present disclosure also provides a system to automatically generate signatures used to detect malware. In one embodiment the system includes an attribute collection module, a heuristic module, a clustering module, a cluster validation module, and a signature generation module. The attribute collection module is used to collect a set of static attributes from a malware dataset and a goodware dataset. The heuristic module is used to build one or more trained or over-trained decision trees from the set of static attributes. The amount of overtraining effects the overall specificity and accuracy of the resulting signatures. The clustering module runs data from an unknown file dataset and a goodware dataset through the one or more decision trees to identify patterns and form one or more clusters. The cluster validation module compares each of the one or more cluster against a reputation value range to determine if each of the one or more clusters is a bad cluster or suspected bad cluster, and the signature creation module creates a signature associated with each bad cluster and each suspected bad cluster. The system may also include a deployment module used to deploy each signature to one or more remote computing systems.

DETAILED DESCRIPTION

Referring now to the figures,FIG. 1is a block diagram of an exemplary signature generation system100for automatically generating signatures based off of static file attributes for detecting malware, while reducing the false positive incident rate. The signature generation system100according to the present application may include one or more signature generator modules110for automatically generating and deploying malware signatures, and data storage modules150. A module can be implemented in hardware, firmware, and/or software.

The signature generator modules are preferably stored on a computer-readable storage medium, loaded into system memory, and executed by a system processor. Security server300, seen inFIG. 2, may be used to execute one or more of the signature generator modules. The data storage modules150store data used by the signature generator modules110to create unique signatures for detecting malware. Examples of such data include malware data set152, goodware data set154, unknown file data set156, current signature dataset160and intermediate data162. The data storage modules150may be implemented in a relational database or another type of database.

The malware dataset152and goodware dataset154contain files, code or software (collectively referred to as “software”) that are known to be malware or goodware, respectively. The malware dataset152and goodware dataset154are continuously updated, but due to the volume of malware generated and detected each day, the historical malware dataset is typically not maintained on a same day-basis. The unknown file dataset156contains software recently collected from the field the status of which is not known such that it may be or contain malware or goodware. New goodware is added to the goodware dataset154. Unknown files are typically collected by security vendors from sources, such as client side computers or servers, or other resources in the field who have recently encountered the software, and determined it to be suspect, but no determination was made as to whether the file is malware or goodware.

As seen inFIG. 1, the one or more signature generator modules110may include an attribute collection module112, a heuristic module114, a clustering module116, a cluster validation module118, a signature creation module120, and a deployment module122.

The attribute collection module112processes the software in the malware dataset152and goodware dataset154looking for physical static attributes associated with the software to build a corpus of static attributes. To represent each static attribute in a suitable form for automatic processing, the attribute collection module assigns each attribute a comparison value. It should be noted, that when clustered, which is described below, an entire cluster of software (or files) can have a numerical value range having “min” and “max” values. The collected attribute data is stored in intermediate data set162within the data storage module150, and is used by the heuristic module114as a training set to build a heuristic model, and used by the clustering module116to mine for patterns and build clusters. A static attribute for software is one or more characteristics of the software that can be ascertained without the need to observe the software during execution. Examples of static attributes include; file size, code size, entropy of file, string characteristics, size of appended data, the section where the software is located within a larger program, and the number of sections in the software. However, the number of static attributes is extensive and this list is in no way intended to limit the static attributes that can be ascertained from software and used with the system of the present application.

The heuristic module114uses the collected static attribute data from the intermediate data set162as a training set to build a heuristics model that is used to identify patterns in the attribute data for clustering. The heuristic model may use decision trees, or other techniques that enable an inference to be drawn as to whether software includes or is malware. Preferably, the heuristic model is a forest of decision trees. Building a forest of decision trees is generally known in the art. However, unlike conventional decision tree forests, the forest of decision trees built in accordance with the present disclosure uses all the collected static attribute data to train the forest of decision trees. Using all collected static attribute data to train the forest of decision trees and specifying that there is no smallest allowable node size is sometimes referred to as over-training the decision tree. Each terminal node in the forest of decision trees is then assigned a number, which are assigned in sequence, between 1 and “N” terminal nodes, where “N” is dependent upon the total number of terminal nodes within the decision tree or forest.

The clustering module116runs the goodware and unknown files from datasets154and156through the heuristic model, which in the example provided is a forest of decision trees, to identify patterns and then organize the patterns so that similar attributes are clustered together. More specifically, each file in the goodware and unknown file datasets154and156is run through the forest of decision trees and the output for the particular file of software is a number sequence in the form of a pattern of “n” unique numbers, where “n” is the number of trees in the forest. This process is repeated for each file in the goodware and unknown files from datasets154and156. The clustering module116then compares each pattern of “n” unique numbers and groups and clusters those that have identical terminal node patterns.

It should be noted that as the collected static attributes are from both the malware dataset152and the goodware dataset154, a resulting cluster can consist of attributes of a single class (e.g., all malware), or can contain a mix of both classes.

Also, number patterns (or sequences) from different instances of a polymorphic malware family tend to cluster together because the attributes of the malware stay consistent despite the polymorphisms. A cluster that contains attributes predominantly indicative of being from known malware (called a “bad cluster”) are presumed to describe behaviors of a single malware family, e.g., a set of malware that are related through polymorphisms, a common code base, or another relationship. Similarly, a cluster that contains attributes predominantly indicative of having bad characteristics but not known to be malware (called a “suspected bad cluster”) are presumed to describe behaviors that are more likely than not of a single malware family. A cluster that contains attributes predominantly indicative of being from goodware (called a “good cluster”) are presumed to describe behaviors of a single goodware family. Similarly, a cluster that contains attributes predominantly indicative of having good characteristics but not known to be goodware (called a “suspected good cluster”) are presumed to describe behaviors that are more likely than not of a single goodware family. By treating the clusters in this fashion, it is possible to reduce the number of false positives because software that falls within a good cluster and software that falls within a suspected good cluster can be treated the same by anti-virus software, i.e., they will not prevent the software from getting into a protected system.

The cluster validation module118analyzes the clusters to validate the quality of each cluster. To validate each cluster, each file in the cluster is assessed for its reputation utilizing an existing classification system. Then the cluster is assessed for purity by looking at the reputations of each file in the cluster.

For each bad cluster and each suspected bad cluster, the signature creation module120then creates a unique signature that can be used to detect malware associated with that cluster. To create the unique signature for a cluster, the signature creation module120conducts a look back or walk back evaluation of each terminal node to find all static attributes involved in making decisions in the forest of decision trees. The signature creation module120then takes the attribute comparison values for all such static attributes in the bad cluster or suspected bad cluster and generates a string in the form of a single complex Boolean expression representing a unique signature for the cluster. This Boolean expression may include inequality strings, such as less than (<) or greater than (>) strings, exact match strings, and set inclusion strings, such as a value is in the set of {a, b, c}. For example, if the forest of decision trees had five decision trees (trees 1-5), where tree 1 had static attribute “a1”, tree 2 had static attribute “a2”, tree 3 had static attribute “a3”, tree 4 had static attribute “a4”, and tree 5 had static attribute “a5”, and if it were determined that the attribute comparison values for static attribute “a1” were between 90 and 100, the attribute comparison values for static attribute “a2” were between 48 and 52, the attribute comparison value for static attribute “a3” was equal to 1000, the attribute comparison value for static attribute “a4” was greater than 150, and the attribute comparison value for static attribute “a5” was in the set of {French, English, German}, the single Boolean expression would be: if a1>=90 && a1<=100 && a2>=48 && a2<=52 && a3==1000 && a4>150 && a5 E {French, English, German}.

The deployment module122is programmed to deploy the unique signatures within a community or enterprise. As seen inFIG. 2, the signature generation system100may be deployed in security server300, which is connected to remote computing systems, e.g., clients210(1)-210(M) via network220. In such a configuration, the deployment module would be configured to transmit or deploy the signatures generated by the system100to the clients.

Using the above system and process, unique signatures for each bad cluster and suspected bad cluster are generated and deployed to the field. On regular intervals, e.g. once a day, new malware may be moved from the new malware dataset may be classified and moved to the historical malware dataset, and new malware dataset is loaded with new unclassified malware. At this point, the system according to the present disclosure may be initiated to repeat the process of generating new signatures for bad and suspected bad clusters.

Although the signature generator modules110are described above and illustrated as separate elements, one or more of modules110(seen inFIG. 1) may represent portions of a single module or application. In certain embodiments, one or more of signature generator modules110may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, one or more of modules110may represent software modules stored and configured to run on one or more computing devices, such as the security server300illustrated inFIGS. 2 and 3, and/or portions of exemplary network architecture400inFIG. 4. One or more of modules110inFIG. 1may also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.

Further, the exemplary signature generation system100ofFIG. 1may be deployed in a variety of ways. For example, all or a portion of the system100may represent portions of a cloud-computing or network-based environment, such as exemplary system200illustrated inFIG. 2. In this exemplary environment, malware signatures are generated by the signature generation system100, and deployed to remote computing systems via the cloud computing or networked-based environment.

FIG. 3is a block diagram of an exemplary security server300capable of implementing one or more of the embodiments described and/or illustrated herein. Security server300broadly represents any single or multi-processor computing device or system capable of executing computer-readable instructions. Examples of security server300include, without limitation, workstations, laptops, client-side terminals, servers, distributed computing systems, handheld devices, or any other computing system or device. In its most basic configuration, the security server300may include at least one processor310, and system memory312.

Processor310generally represents any type or form of processing unit capable of processing data or interpreting and executing instructions. In certain embodiments, processor310may receive instructions from a software application or module. These instructions may cause processor310to perform the functions of one or more of the exemplary embodiments described and/or illustrated herein. For example, processor310may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the functions described above for the modules110, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions. Processor310may also perform and/or be a means for performing any other steps, methods, or processes described and/or illustrated herein.

System memory312generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or other computer-readable instructions. Examples of system memory312include, without limitation, random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory device. Although not required, in certain embodiments security server300may include both a volatile memory unit (such as, for example, system memory312) and a non-volatile storage device (such as, for example, primary storage device352, as described in detail below). In one example, one or more of modules110and150fromFIG. 1may be loaded into system memory312. In another example, one or more of modules110fromFIG. 1may be loaded into system memory312, and the data storage modules150fromFIG. 1may be loaded into primary storage device352, as shown by dotted lines inFIG. 3.

In certain embodiments, exemplary security server300may also include one or more components or elements in addition to processor310and system memory312. For example, as illustrated inFIG. 3, security server300may include a memory controller316, an Input/Output (I/O) controller318, and a communication interface320, each of which may be interconnected via a communication infrastructure322. Communication infrastructure322can be any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communication infrastructure322include, without limitation, a communication bus (such as an ISA, PCI, PCIe, or similar bus) and a network.

Memory controller316can be any type or form of device capable of handling memory or data or controlling communication between one or more components of security server300. For example, in certain embodiments memory controller316may control communication between processor310, system memory312, and I/O controller318via communication infrastructure322. Memory controller316may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps, features or functions described and/or illustrated herein, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions.

I/O controller318can be any type or form of device capable of coordinating and/or controlling the input and output functions of a computing device. For example, I/O controller318may control or facilitate transfer of data between one or more elements of security server300, such as processor310, system memory312, communication interface320, display adapter330, input interface340, and storage interface350. I/O controller318may be used, for example, to perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps, features or functions described and/or illustrated herein, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions. I/O controller318may also be used to perform and/or be a means for performing other steps and features set forth in the instant disclosure.

Communication interface320can be any type or form of communication device or adapter capable of facilitating communication between exemplary security server300and one or more additional devices. For example, communication interface320may facilitate communication between security server300and a private or public network including additional computing systems. Examples of communication interface320include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), and/or a modem. In at least one embodiment, communication interface320provides a direct connection to a remote computing system (e.g., a client computing device or a server) via a direct link to a network, such as the Internet. Communication interface320can also be configured to indirectly provide such a connection through, for example, a local area network (such as an Ethernet network), a personal area network, a telephone or cable network, a cellular telephone connection, a satellite data connection, or any other suitable connection.

The communication interface320can also function as a host adapter configured to facilitate communication between security server300and one or more additional network or storage devices via an external bus or communications channel. Examples of host adapters include, without limitation, SCSI host adapters, USB host adapters, IEEE 1394 host adapters, SATA and eSATA host adapters, ATA and PATA host adapters, Fibre Channel interface adapters, Ethernet adapters, or the like. Communication interface320may also allow the security server300to engage in distributed or remote computing. For example, communication interface320can receive instructions from a remote device or send instructions to a remote device for execution. In certain embodiments, communication interface320may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps, features or functions described and/or illustrated herein, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions. Communication interface320may also be used to perform and/or be a means for performing other steps, features and/or functions set forth in the instant disclosure.

Continuing to refer toFIG. 3, security server300may also include at least one display device332coupled to communication infrastructure322via a display adapter330. Display device332can be any type or form of device capable of visually displaying information forwarded by display adapter330. Similarly, display adapter330can be any type or form of device configured to forward graphics, text, and other data from communication infrastructure322(or from a frame buffer, as known in the art) for display on display device332.

The security server300may also include at least one input device342coupled to communication infrastructure322via an input interface340. Input device342can be any type or form of input device capable of providing input, either computer or human generated, to the computing system300. Examples of input device342include, without limitation, a keyboard, a pointing device, a speech recognition device, or any other input device. In at least one embodiment, input device342may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps, features or functions described and/or illustrated herein, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions. Input device342may also be used to perform and/or be a means for performing other steps, features and/or functions set forth in the instant disclosure.

As noted above, the security server300may also include a primary storage device352and a backup storage device354coupled to communication infrastructure322via a storage interface350. Storage devices352and354can be any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. For example, the storage devices352and354may be a magnetic disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash drive, or the like. Storage interface350can be any type or form of interface or device for transferring data between storage devices352and354and other components of security server300. In one example, the data storage modules150fromFIG. 1may be stored in primary storage device352.

In certain embodiments, storage devices352and354may be configured to read from and/or write to a removable storage unit configured to store computer software, data, or other computer-readable information. Examples of suitable removable storage units include a floppy disk, a magnetic tape, an optical disk, a flash memory device, or the like. Storage devices352and354may also include other similar structures or devices for allowing computer software, data, or other computer-readable instructions to be loaded into security server300. For example, storage devices352and354may be configured to read and write software, data, or other computer-readable information. Storage devices352and354may also be a part of the security server300or may be a separate device accessed through other interface systems.

In certain embodiments, storage devices352and354may be used, for example, to perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps, features or functions described and/or illustrated herein, including the attribute collection, heuristic modeling, clustering, cluster validation, signature creation and deploying functions. Storage devices352and354may also be used to perform and/or be a means for performing other steps, features and/or functions set forth in the instant disclosure.

Many other devices or subsystems may be connected to security server300. Conversely, all of the components and devices illustrated inFIG. 3need not be present to practice the system100of the present disclosure. Security server300may also employ any number of software, firmware, and/or hardware configurations. For example, one or more of the exemplary embodiments of the system100may be encoded as a computer program (also referred to as computer software, software applications, computer-readable instructions, or computer control logic) on a computer-readable medium.

The computer-readable medium containing the computer program may be loaded into security server300. All or a portion of the computer program stored on the computer-readable medium may then be stored in system memory312and/or various portions of storage devices352and354. When executed by processor310, a computer program loaded into security server300may cause processor310to perform and/or be a means for performing the functions of one or more of the exemplary embodiments described and/or illustrated herein. Additionally or alternatively, one or more of the exemplary embodiments described and/or illustrated herein may be implemented in firmware and/or hardware.

As noted above, the signatures generated by the system100may be deployed to remote computing systems.FIG. 4is a block diagram of an exemplary network architecture400in which client systems410,420, and430and servers440and450may be coupled to a network460. In the exemplary embodiment ofFIG. 4, the system100of the present disclosure is loaded into client system410. However, the system100of the present disclosure may be loaded into any client system or server connected to the network460. Client systems410,420, and430can be any type or form of computing device or system, such as exemplary security server300inFIG. 3. Similarly, servers440and450can be computing devices or systems, such as application servers or database servers, configured to provide various database services and/or run certain software applications. Network460can be any telecommunication or computer network including, for example, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), or the Internet.

As illustrated inFIG. 4, one or more storage devices470(1)-(N) may be directly attached to server440. Similarly, one or more storage devices480(1)-(N) may be directly attached to server450. Storage devices470(1)-(N) and storage devices480(1)-(N) can be any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. In certain embodiments, storage devices470(1)-(N) and storage devices480(1)-(N) may represent network-attached storage (NAS) devices configured to communicate with servers440and450using various protocols, such as NFS, SMB, or CIFS.

Servers440and450may also be connected to a storage area network (SAN) fabric490. SAN fabric490can be any type or form of computer network or architecture capable of facilitating communication between a plurality of storage devices. SAN fabric490may facilitate communication between servers440and450and a plurality of storage devices492(1)-(N) and/or an intelligent storage array494. SAN fabric490may also facilitate, via network460and servers440and450, communication between client systems410,420, and430and storage devices492(1)-(N) and/or intelligent storage array494in such a manner that devices492(1)-(N) and array494appear as locally attached devices to client systems410,420, and430. As with storage devices470(1)-(N) and storage devices480(1)-(N), storage devices492(1)-(N) and intelligent storage array494generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions.

In certain embodiments, and with reference to the exemplary security server300ofFIG. 3, a communication interface, such as communication interface320(seen inFIG. 3), may be used to provide connectivity between each client system410,420, and430and network460. Client systems410,420, and430may be able to access information on server440or450using, for example, a web browser or other client software. Such software may allow client systems410,420, and430to access data hosted by server440, server450, storage devices470(1)-(N), storage devices480(1)-(N), storage devices492(1)-(N), or intelligent storage array494.

Referring now toFIG. 5, an exemplary flow diagram of the steps performed by one embodiment of the system100is disclosed. Initially, at step510, one or more datasets are established. Preferably, the datasets include the malware dataset152, the goodware dataset154, and the unknown file dataset156. At step520, the security server300executes the attribute collection module to collect attributes from the malware dataset152and goodware dataset154. At step530, using the collected attribute data, the security server300initiates the heuristic module114to build a heuristic model that is used to identify patterns in the attribute data for clustering. At step540, the security server300executes the clustering module116to run goodware and unknown files from datasets154and156through the heuristic model to identify patterns and then organize the patterns so that files with similar attributes are clustered together. At step550, the security server300executes the clustering validation module116to validate the clusters against reputation values, as described above. At step560, clusters determined to be bad or suspected bad clusters are processed by the security server300executing the signature creation module120to create a signature representative of the bad or suspected bad cluster, as described above. At step570, once signatures are created for all bad and suspected bad clusters they are deployed by security server300executing the deployment module122.