Detecting polymorphic threats

A polymorphic threat manager monitors an incoming email stream, and identifies incoming email messages to which executable files are attached. The polymorphic threat manager characterizes incoming executable files according to at least one metric. For example, the polymorphic threat manager can decompose an executable file into fragments, hash some or all of these, and use the hashes as characterization metrics. The polymorphic threat manager subsequently de-obfuscates executable files, and creates corresponding characterization metrics for the de-obfuscated images. The characterizations of executable files before and after de-obfuscation are compared, and if they differ sufficiently, the polymorphic threat manager determines that the file in question is polymorphic. The characterization metrics of such an executable file after de-obfuscation can be used as a signature for that file.

TECHNICAL FIELD

This invention pertains generally to computer security, and more specifically to robustly detecting and generating signatures for polymorphic malicious code.

BACKGROUND

Mass-mailing worms are some of the most prevalent and troublesome threats to Internet users today. Worms like Netsky, Beagle, MyDoom, and most recently, Sober, have caused millions of dollars in damage and cleanup costs. To make matters worse, the increasing availability and quality of runtime packers and other obfuscation tools are making it easier for worm writers to automate the creation of new variants of a worm, making analysis more complicated and time consuming.

Generated signatures can be utilized in order to detect and block malicious code. However, existing signature generation methodologies do not account for oligomorphic or polymorphic malicious executable images, which can change their external form each time they replicate. The existing signature generation methods do not detect the fact that these different forms are instantiations of the same worm. Therefore, such methods create a different signature for each new replica of the worm. This can overwhelm any agent (such as a centralized correlation server) processing the detection and management of malicious code.

What is needed are methods, systems and computer readable media for generating robust signatures that can commonly identify a polymorphic worm in its various forms.

DISCLOSURE OF INVENTION

Computer-implemented methods, computer systems and computer-readable media manage polymorphic malicious code. A polymorphic threat manager monitors an incoming email stream, and identifies incoming email messages to which executable files are attached. The polymorphic threat manager characterizes incoming executable files according to at least one metric. For example, the polymorphic threat manager can decompose an executable file into fragments, hash some or all of these, and use the hashes as characterization metrics. The polymorphic threat manager subsequently de-obfuscates executable files, and creates corresponding characterization metrics from the de-obfuscated file images. The characterizations of executable files before and after de-obfuscation are compared, and if they differ sufficiently, the polymorphic threat manager determines that the file in question is polymorphic. The characterization metrics of such an executable file after de-obfuscation can be used as a signature for that file.

These automatically generated characterization metrics can be used as the input to a larger, distributed correlation system. In such scenarios, after identifying a polymorphic executable file, the polymorphic threat manager submits only de-obfuscated characterization metrics to the correlation system. This filtration reduces the load on the correlation system, which could otherwise be overwhelmed by attempts to correlate the huge number of unrelated characterization metrics that would be generated from an obfuscated polymorphic image.

The polymorphic threat manager can also compare characterizations of de-obfuscated executable files to characterizations of the de-obfuscated images of known malicious polymorphic entities, and where they are substantially similar, determine that the executable file comprises that malicious polymorphic entity. More specifically, the polymorphic threat manager can compare characterizations of executable files after de-obfuscation to characterizations of stored de-obfuscated images of executable files received earlier and determined to be polymorphic. Responsive to the characterization of an executable file after de-obfuscation being sufficiently similar to the stored characterization of a de-obfuscated executable file known to be polymorphic, the polymorphic threat manager concludes that the two executable files comprise different forms of a single polymorphic executable file.

DETAILED DESCRIPTION

FIG. 1illustrates a high level overview of a system100for practicing some embodiments of the present invention. A polymorphic threat manager101detects and generates signatures103for polymorphic malicious code105. It is to be understood that although the polymorphic threat manager101is illustrated as a single entity, as the term is used herein a polymorphic threat manager101refers to a collection of functionalities which can be implemented as software, hardware, firmware or any combination of these. Where a polymorphic threat manager101is implemented as software, it can be implemented as a standalone program, but can also be implemented in other ways, for example as part of a larger program, as a plurality of separate programs, as one or more device drivers or as one or more statically or dynamically linked libraries.

One of the distinguishing characteristics of polymorphic threats105is that they typically decrypt themselves in order to execute the actual viral body. Typically, this viral body remains fixed, though the outward appearance of the virus or worm might change from generation to generation due to the re-encryption of the body. The polymorphic threat manager101exploits the tell-tale decryption behavior to automate the identification of polymorphic executables105, and when possible, to extract a signature103from the viral bodies.

The polymorphic threat manager101, which can operate, for example on an email gateway107as illustrated, watches all incoming email109, and extracts all executable attachments111for further analysis. The polymorphic threat manager101takes one or more baseline metric(s)113of each extracted executable file111. In one embodiment of the present invention, the metric113is in the form of one or more baseline hashes115of the unprocessed executable attachment111. In such an embodiment, typically, an executable file111is decomposed into various fragments117, and then a hash115of each fragment117is computed. The polymorphic threat manager101can compute the hash115by applying any suitable hashing algorithm, such as CRC, MD5, or SHA-1. The polymorphic threat manager101can decompose the file111into one or more pieces (not illustrated) based on any consistent criteria, such as identifying sections within the executable format (PE format on Windows, ELF on Linux, etc.).

In some embodiments, the hashes115(or other metric113type) of the executable file111are compared to a pre-computed list119of signatures105of known benign executables121. All metrics113that match an entry on the list119of known benign executables121are adjudicated to be themselves associated with legitimate executables121, and are not further processed.

Turning now toFIG. 2, it is to be understood that in other embodiments an executable file111can be characterized according to metrics113other than hashes thereof115. As illustrated inFIG. 2, in some embodiments the polymorphic threat manager101characterize the executable file111by running it, e.g., in an emulator (not illustrated) or a virtual machine201, dumping the resulting memory image203and using that as a characterization metric113. The polymorphic threat manager101can also characterize the executable file111by running it and tracking instruction usage, recording a control flow graph of at least one section thereof, noting a size, range or entropy of at least a part of at least one section, detecting a transformation of code or data, or detecting the execution of one or more instructions, or the absence thereof. Any of these data can be used as characterization metrics113, and it is to be understood that in various embodiments, the polymorphic threat manager101can utilizes any of these or other metrics113as well as combinations thereof in order to characterize executable files111.

Returning now toFIG. 1, the polymorphic threat manager101subsequently passes the executable file111through one or more de-obfuscation techniques. For example, the polymorphic threat manager101can utilize an unpacker122to remove any runtime packing and/or compression from the file111. The polymorphic threat manager101can also run the executable111in an emulator or virtual machine201, dump the memory203after detecting decryption or after a fixed amount of time (not illustrated inFIG. 1), and use that as an image123of the file111in its de-obfuscated form. These techniques remove compression and encryption in the executable file111, which can be indicators of a polymorphic threat105. By manipulating the executable111into its decrypted, decompressed state, the polymorphic threat manager101can better analyze the file111and generate a signature103therefrom.

In some embodiments, the polymorphic threat manager101de-obfuscates executable files111by canonicalizing the instructions therein. Malicious code105can obfuscate its function by using non-standard or superfluous instructions, or by using more, fewer, or unexpected registers or similar techniques. By standardizing code, the polymorphic threat manager101can identify the function thereof, and thus unearth, process and create a single signature103for different manifestations of a single polymorphic threat105.

After de-obfuscation, the same characterization process(es) as described above are applied to the de-obfuscated image123obtained from the executable file111. In some embodiments, the metrics113of the de-obfuscated executable123are compared to a pre-computed list119of signatures103from known legitimate executables121as described above. As described above, metrics113that match a list119entry are assumed legitimate and not employed in subsequent processing.

The two sets of characterization metrics113(pre and post de-obfuscation) are compared, with any differences between the two indicating that the executable file111might be obfuscated and thus polymorphic. If the characterization113of a de-obfuscated image123is sufficiently different from those of the pre de-obfuscation executable file111, the executable111is adjudicated to be polymorphic. The de-obfuscated image123can be stored locally for further analysis, and can also be reported to a centralized component such as a remote correlation server125. A system in which a central correlation server125is utilized in the correlation of malicious code across a network is described in co-pending U.S. patent application Ser. No. 11/214,631, titled “Detection of E-mail Threat Acceleration,” filed on Aug. 30, 2005, having the same inventors and assignee, the entirety of which is herein incorporated by reference. It is to be understood that in some but not all embodiments of the present invention, a plurality of polymorphic threat managers101are deployed at e-mail gateways107across a network, each of which supplies signatures103and other information concerning detected polymorphic threats105to a correlation server125, as per the Detection of Email Threat Acceleration application.

The characterizations113of executable files111found to be polymorphic can be compared to characterizations113(signatures103) of known polymorphic threats105, in order to determine whether the executable file111under analysis comprises one of these. The characterizations113of de-obfuscated image123can also be compared to characterizations113of de-obfuscated images123of other polymorphic executables111detected in the same manner at the e-mail gateway107by the polymorphic threat manager101. If two or more different executable files111have the same de-obfuscated characterizations113but have different baseline (pre de-obfuscated) characterizations113, then the attachments are likely different forms of the same polymorphic threat105, and are so adjudicated to be. In such a case, the polymorphic threat manager101stores the common de-obfuscated characterization113locally for use as a signature103for that polymorphic threat105.

In embodiments in which the malicious threat manager101submits information to a correlation server125, when the polymorphic threat manager101determines that a given incoming executable attachment111has one of the locally stored de-obfuscated characterizations113, then the polymorphic threat manager101only sends the common characterizations113, and not the remaining metrics113(e.g., uncommon hashes115) for the file111. This is because the remaining metrics113are for the de-obfuscated polymorphic body of the threat rather than the viral portion, and are not valuable for correlation)

Rather than overloading the correlation server125with a large number of unique and potentially useless metrics113, in some embodiment the polymorphic threat manager selects only the metrics113(e.g., hashes115) that are most likely to be successfully correlated. Intelligent metric113selection at the malicious threat manager101can achieve a decrease in bandwidth and processing at the correlation server125by an order of magnitude or more. In such embodiments, the malicious threat manager101can comprise an integral component in a robust, scalable correlation infrastructure capable of coping with massive outbreaks of polymorphic worms105. Since each instance of a polymorphic worm105may have a different set of metrics113, simply forwarding all of the metrics113for each executable attachment111to the correlation server125can cause a flood of different metrics113to be reported to the system, potentially resulting in a denial of service situation.

As will be understood by those familiar with the art, the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the modules, managers, functions, layers, features, attributes, methodologies and other aspects are not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, divisions and/or formats. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, managers, functions, layers, features, attributes, methodologies and other aspects of the invention can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present invention is implemented as software, the component can be implemented as a script, as a standalone program, as part of a larger program, as a plurality of separate scripts and/or programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.