Cyber-security framework for application of virtual features

A non-transitory storage medium having stored thereon logic wherein the logic is executable by one or more processors to perform operations is disclosed. The operations may include parsing an object, detecting one or more features of a predefined feature set, evaluating each feature-condition pairing of a virtual feature using the one or more values observed of each of the one or more detected features, determining whether results of the evaluation of one or more feature-condition pairings satisfies terms of the virtual feature, and responsive to determining the results of the evaluation satisfy the virtual feature, performing one or more of a static analysis to determine whether the object is associated with anomalous characteristics or a dynamic analysis on the object to determine whether the object is associated with anomalous behaviors.

FIELD

Embodiments of the disclosure relate to the field of cyber security. More specifically, embodiments of the disclosure relate to a malware detection technique for performing an analysis of a group of objects, an object or a sub-object to determine whether further analysis should be performed on the group of objects, the object or the sub-object.

GENERAL BACKGROUND

Current network security appliances receive large amounts of data to analyze, possibly prior to passing to one or more endpoint devices. The large amounts of data make timely analysis difficult as analyzing the entire amount of received data is time-consuming and processing-intensive. Therefore, current network security appliances attempt to employ a filtering process that acts to determine whether portions of the received data exhibit some characteristics of suspiciousness prior to performing an in-depth analysis on the one or more portions of the received data.

However, current filtering techniques are reactive and attempt to match extracted features or other indicators of known malware with a portion of the received data in order to determine whether the portion of received data is suspicious. Specifically, malware detection systems within network security appliances often parse received data attempting to identify and then match a feature of the received data with an indicator of known malware to assess suspiciousness or even maliciousness. This technique often leads to false positives as benign (e.g., non-malicious) data may share an indicator with malware. As a result, in-depth malware analysis may be performed on the benign data, wasting time and requiring unnecessary processing. Of course, where the indicator is a fingerprint or signature in the form of a hash of known malware, the chance of a match with benign data is greatly reduced, but such a fingerprint or signature is often not available (e.g., as is the case with ‘zero day’ malware).

Many different malware detection approaches can suffer the effects of false positives. Known Intrusion Protection Systems (IPS) and Intrusion Detection Systems (IDS) are noteworthy for producing an enormous rate of false positives. Known “two-stage” advanced malware detection systems deployed at the periphery of a network employ a first static analysis stage used to filter incoming data before submitting that portion deemed suspicious for second-stage dynamic analysis via execution in a virtual environment. In such systems, the static analysis is configured to avoid false negatives and, as a trade-off, can unfortunately tag benign data as suspicious and then submit those false positives for dynamic analysis.

Additionally, received data may contain one or more features that seem benign in an isolated manner but, combined with other seemingly suspicious, malicious or benign features, are actually part of a malicious cyber-attack.

DETAILED DESCRIPTION

Various embodiments of the disclosure relate to a real-time traffic analysis that determines whether a portion of received data should be provided to a static analysis logic and/or a dynamic analysis logic for an in-depth analysis. Such a real-time traffic analysis improves detection of exploits, malware attacks and/or anomalous characteristics/behaviors as the initial analysis performed by a real-time traffic analysis logic determines whether at least a portion of the received data has at least a first level of suspiciousness and/or maliciousness to warrant an in-depth analysis by either a static analysis logic and/or a dynamic analysis logic.

Real-time traffic analysis seeks to utilize extracted features of malware (which may themselves provide an indication that the portion of received data is suspicious and/or malicious), and/or seemingly benign features (which in isolation may not provide an indication that the portion of received data is suspicious and/or malicious) to create a more tailored, or directed, analysis. Real-time traffic analysis utilizes one or more “virtual features,” which may be each a combination of one or more features, typically expressed as a logical expression. Additionally, a condition may be applied to each feature within the virtual feature. A condition applied to a feature seeks to limit the values evaluated during real-time traffic analysis to those that, through experiential knowledge and/or machine learning, are known to be suspicious and/or malicious (the application of a condition to a feature may be referred to as a feature-condition pairing). Thus, the combination of feature-condition pairings within a logical expression (e.g., feature-condition pairings separated by logical operators comprising a virtual feature) leads to a more tailored analysis. The use of one or more virtual features to determine whether a portion of received data exhibits at least a first threshold of suspiciousness, maliciousness and/or anomalous characteristics and/or behaviors prior to performing an in-depth static and/or dynamic analysis, leads to fewer false positives. Thus, real-time traffic analysis enables a malware/threat detection system (e.g., deployed within a network appliance) to determine whether a portion of data exhibits at least a first threshold of suspiciousness, maliciousness and/or anomalous characteristics and/or behaviors more efficiently and accurately than pre-filter mechanisms that merely employ feature and/or signature matching techniques.

In one embodiment, the threat detection system receives an object to analyze. Subsequently, the object is passed to a real-time traffic analysis (RTTA) logic within the threat detection system. The RTTA logic parses the object looking for features of a predefined feature set. During parsing, the RTTA logic maintains a representation of the detected features of the predefined feature set as “observations,” and the value of each instance of the observation, if applicable. In one embodiment, the representation of values of detected features maintained by the RTTA logic may be a value for every instance of a detected feature (e.g., a size value for every embedded object) or a count of the number of instances of a feature (e.g., the number of JavaScript instances, embedded objects, Universal Record Locators (URLs) or other features that may appear multiple times within an object or portion of network traffic). In a second embodiment, the representation of detected features maintained by the RTTA logic may be a subset of the values of every instance of a detected feature. In such an embodiment, the RTTA logic may maintain the representation of detected features by eliding certain, similar values (e.g., omit duplicate values and/or merge similar values into ranges) of a detected feature. Upon completion of parsing, the representation of the values of detected features may be recorded in a storage device for accessing during an evaluation performed by the RTTA logic.

Once the object has been parsed and any values of observations have been recorded, the RTTA logic evaluates each feature-condition pairing set forth in the virtual feature against the value, or set of all values of the corresponding observation. As is defined below, the virtual feature is a combination of one or more feature-condition pairings, wherein the combination may be represented as a logical expression. Thus, in instances in which an observation has a value, the set of all values of an observation is paired with one or more conditions. In some embodiments, depending on the content of the received data, an observation may have a plurality of values (e.g., when observationX, for example, is the size of an attachment to an email, observationXwill have a plurality of values when, for example, an email under analysis includes a plurality of attachments, wherein at least a first attachment has a different size than a second attachment).

Following the evaluation of each feature-condition pairing against the value, or set of all values of the corresponding observation, the RTTA logic generates an observation vector based on the feature-condition pairing evaluation and determines whether the observation vector is a member of the truth set corresponding to the virtual feature. A notification that the virtual feature was observed is provided to a RTTA score determination logic within the threat detection system. The threat detection system then generates a score indicating the level of suspiciousness and/or maliciousness of the portion of analyzed received data. In one embodiment, when the score is above a first threshold, the reporting logic may generate an alert that the object is malicious. When the score is greater than a second threshold but lower than the first threshold, the object may be provided to the static analysis logic and/or the dynamic analysis logic for further analysis. When the score is less than the second threshold, the threat detection system may determine no further analysis is needed (e.g., the object is benign).

The use of one or more virtual features by a threat detection system enables several advantages over current systems by providing a more efficient and flexible analysis. Specifically, the analysis performed by the RTTA using one or more virtual features enables the RTTA to combine observable features with conditions derived via experiential knowledge and/or machine learning to target particular exploits, vulnerabilities or malware. Additionally, an analysis including evaluating feature-condition pairings of a virtual feature using at least one or more values of a representative set of observed values within an object provides increased efficiency by eliminating the need to analyze all observed values using a rule-based detection scheme.

In the following description, certain terminology is used to describe features of the invention. For example, in certain situations, both terms “logic” and “engine” are representative of hardware, firmware and/or software that is configured to perform one or more functions. As hardware, logic (or engine) may include circuitry having data processing or storage functionality. Examples of such circuitry may include, but are not limited or restricted to a microprocessor, one or more processor cores, a programmable gate array, a microcontroller, a controller, an application specific integrated circuit, wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic.

Logic (or engine) may be software in the form of one or more software modules, such as executable code in the form of an executable application, an application programming interface (API), a subroutine, a function, a procedure, an applet, a servlet, a routine, source code, object code, a shared library/dynamic link library, or one or more instructions. These software modules may be stored in any type of a suitable non-transitory (computer-readable) storage medium, or transitory storage medium (e.g., electrical, optical, acoustical or other form of propagated signals such as carrier waves, infrared signals, or digital signals). Examples of non-transitory storage medium may include, but are not limited or restricted to a programmable circuit; a semiconductor memory; non-persistent storage such as volatile memory (e.g., any type of random access memory “RAM”); persistent storage such as non-volatile memory (e.g., read-only memory “ROM”, power-backed RAM, flash memory, phase-change memory, etc.), a solid-state drive, hard disk drive, an optical disc drive, or a portable memory device. As firmware, the executable code is stored in persistent storage.

According to one embodiment, the term “malware” may be construed broadly as any code or activity that initiates a malicious attack and/or operations associated with anomalous or unwanted behavior. For instance, malware may correspond to a type of malicious computer code that executes an exploit to take advantage of a vulnerability, for example, to harm or co-opt operation of a network device or misappropriate, modify or delete data. Malware may also correspond to an exploit, namely information (e.g., executable code, data, command(s), etc.) that attempts to take advantage of a vulnerability in software and/or an action by a person gaining unauthorized access to one or more areas of a network device to cause the network device to experience undesirable or anomalous behaviors. The undesirable or anomalous behaviors may include a communication-based anomaly or an execution-based anomaly, which, for example, could (1) alter the functionality of an network device executing application software in an atypical manner (a file is opened by a first process where the file is configured to be opened by a second process and not the first process); (2) alter the functionality of the network device executing that application software without any malicious intent; and/or (3) provide unwanted functionality which may be generally acceptable in another context. Additionally, malware may be code that initiates unwanted behavior which may be, as one example, uploading a contact list to cloud storage without receiving permission from the user. For convenience, the terms “malware” and “exploit” shall be used interchangeably herein unless the context requires otherwise.

The term “processing” may include launching an application wherein launching should be interpreted as placing the application in an open state and performing simulations of actions typical of human interactions with the application. For example, the application, an Internet browsing application may be processed such that the application is opened and actions such as visiting a website, scrolling the website page, and activating a link from the website are performed (e.g., the performance of simulated human interactions).

The term “network device” may be construed as any electronic device with the capability of connecting to a network, downloading and installing mobile applications. Such a network may be a public network such as the Internet or a private network such as a wireless data telecommunication network, wide area network, a type of local area network (LAN), or a combination of networks. Examples of a network device may include, but are not limited or restricted to, a laptop, a mobile phone, a tablet, etc. Herein, the terms “network device,” “endpoint device,” and “mobile device” will be used interchangeably. The terms “mobile application” and “application” should be interpreted as software developed to run specifically on a mobile network device.

The term “malicious” may represent a probability (or level of confidence) that the object is associated with a malicious attack or known vulnerability. For instance, the probability may be based, at least in part, on (i) pattern matches; (ii) analyzed deviations in messaging practices set forth in applicable communication protocols (e.g., HTTP, TCP, etc.) and/or proprietary document specifications (e.g., Adobe PDF document specification); (iii) analyzed compliance with certain message formats established for the protocol (e.g., out-of-order commands); (iv) analyzed header or payload parameters to determine compliance, (v) attempts to communicate with external servers during processing in one or more VMs, (vi) attempts to access memory allocated to the application during virtual processing, and/or other factors that may evidence unwanted or malicious activity.

The term “virtual feature” may be interpreted as a combination of one or more feature-condition pairings, where the combination may be suspicious, malicious or unwanted. Additionally, a first virtual feature may be used as a feature within a second virtual feature. In one embodiment, the combination may be expressed as a logical expression having one or more “terms,” wherein a term may include one or more feature-condition pairings.

The term “feature” may be interpreted as a static trait that may be included in or exhibited by an object, group of objects or sub-object. Examples of features include, but are not limited or restricted to, a size of an object or sub-object, presence of an embedded object, size of an embedded object, number of embedded objects, presence of a URL in the object, number of URLs in the object, presence of a predefined signature, adherence or non-adherence to a predefined rule or protocol.

The term “observation” may be interpreted as features actually observed/detected within object, sub-object, etc. (for example, during or after parsing once the incoming traffic has been received).

The term “condition” may be interpreted as whether a predefined characteristic is present within the portion of network traffic being analyzed (e.g., a flow, a group of objects, an object, and/or a sub-object). Additionally, a condition may be a numeric evaluation such that a determination is made as to whether the set of all observed values of the feature include a particular number, intersect a given range, and/or contain any members greater than or less than a predefined threshold.

The term “observation vector” may be interpreted as a representation of the results of an evaluation of each condition with its corresponding feature, as evaluated for each feature-condition pairing of a single, virtual feature. In one embodiment, such a representation may take the form of a binary construct which may be understood as a vector or a one dimensional array. Herein, the term “observation vector” will be used but the use is not intended to limit the scope of the disclosure. In one example, a condition may be the presence of a feature. In a second example, a condition may be the size of the feature is greater than or equal to a predefined threshold. A condition may be applied to more than one feature (e.g., determining the presence of more than one feature) and/or a feature may be evaluated by more than one feature-condition pairing (e.g., a first term, as described below, of the virtual feature may determine whether a feature is greater than a first threshold and a second term of the virtual feature may determine whether the feature is less than or equal to a second threshold).

The term “truth set” may be interpreted as a representation of the one or more possible results of the evaluation of the feature-condition pairings of a virtual feature based on the values of the observations (e.g., detected features) that satisfy the terms of the corresponding virtual feature, where there is one truth set per virtual feature. The representation of the truth set may take the form of, for example, a set of binary vectors, wherein each binary vector represents a result of an evaluation of the feature-condition pairings of the virtual feature. All vectors included in the truth set represent solutions to the corresponding virtual feature.

The invention may be utilized by a cyber-security platform for performing a real-time traffic analysis of a group of objects, an object and/or a sub-object to determine whether the group of objects, the object and/or the sub-object is to be provided to a static analysis logic and/or a dynamic analysis logic for further analysis as to whether the group of objects, the object and/or the sub-object is suspicious, malicious or unwanted. As this invention is susceptible to embodiments of many different forms, it is intended that the present disclosure is to be considered as an example of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described.

II. Network Appliance Deployment

Referring toFIG.1, an exemplary block diagram of a threat detection system110including a real-time traffic analysis logic113deployed within the network100is shown. In one embodiment, the network100may be an enterprise network that includes the threat detection system110, a router160, an optional firewall161, a network switch162, and one or endpoint devices163. The network100may include a public network such as the Internet, a private network (e.g., a local area network “LAN”, wireless LAN, etc.), or a combination thereof. The router160serves to receive data, e.g., packets, transmitted via a wireless medium (e.g., a Wireless Local Area Network (WLAN) utilizing the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard) and/or a wired medium from the cloud computing services170and the endpoint devices163. As is known in the art, the router160may provide access to the Internet for devices connected to the network110.

In one embodiment, the network switch162may capture network traffic, make a copy of the network traffic, pass the network traffic to the appropriate endpoint device(s)163and pass the copy of the network traffic to the threat detection system110. In a second embodiment, the network switch162may capture the network traffic and pass the network traffic to the threat detection system110for processing prior to passing the network traffic to the appropriate endpoint device(s)163. In such an embodiment, the network traffic will only be passed to the appropriate endpoint device(s)163if the analysis of the network traffic does not indicate that the network traffic is associated with malicious, anomalous and/or unwanted characteristics and/or behaviors.

The threat detection system110includes a communication interface111, a storage device112, a real-time traffic analysis (RTTA) logic113, a RTTA score determination logic114, a static analysis logic120, a dynamic analysis logic130, a classification logic140, and a reporting logic150.

As shown, the threat detection system110is communicatively coupled with the cloud computing services170, the Internet and one or more endpoint devices163via the communication interface111, which directs at least a portion of the network traffic to the RTTA logic113. The RTTA logic113includes one or more modules that, when executed by one or more processors, performs a real-time traffic analysis on the portion of the network traffic by analyzing the portion of the network traffic in light of one or more features and/or virtual features. When the real-time traffic analysis results in a score that is greater than a first threshold, the portion of the network traffic is provided to the static analysis logic120and/or the dynamic analysis130. The static analysis logic120may include one or more software modules that, when executed by one or more processors, performs static scanning on a particular object, namely heuristics, exploit signature checks and/or vulnerability signature checks for example. The static analysis logic120and the dynamic analysis logic130may be one or more software modules executed by the same processor or different processors, where these different processors may be located within the same processor package (e.g., different processor cores) and/or located at remote or even geographically remote locations that are communicatively coupled (e.g., by a dedicated communication link) or a network.

More specifically, as shown, static analysis logic120may be configured with heuristics logic121, exploit matching logic122, and/or vulnerability matching logic123. Heuristics logic121is adapted for analysis of certain portions of an object under analysis (e.g., the object may include a binary file) to determine whether any portion corresponds to either (i) a “suspicious” identifier such as either a particular Uniform Resource Locator “URL” that has previously been determined as being associated with known exploits, a particular source or destination (IP or MAC) address that has previously been determined as being associated with known exploits; or (ii) a particular exploit pattern. When deployed, the exploit matching logic122may be adapted to perform exploit signature checks, which may involve a comparison of an object under analysis against one or more pre-stored exploit signatures (e.g., pre-configured and predetermined attack patterns) from signatures database124. Additionally or in the alternative, the static analysis logic120may be configured with vulnerability matching logic123that is adapted to perform vulnerability signature checks, namely a process of uncovering deviations in messaging practices set forth in applicable communication protocols (e.g., HTTP, TCP, etc.). The term “signature” designates an indicator of a set of characteristics and/or behaviors exhibited by one or more exploits that may not be unique to those exploit(s). Thus, a match of the signature may indicate to some level of probability, often well less than 100%, that an object constitutes an exploit. In some contexts, those of skill in the art have used the term “signature” as a unique identifier or “fingerprint,” for example, of a specific virus or virus family (or other exploit), which is generated for instance as a hash of its machine code, and that is a special sub-case for purposes of this disclosure.

The classification logic140may be configured to receive the static-based results (e.g., results from static analysis, metadata associated with the incoming network traffic, etc.), the VM-based results and/or the RTTA analysis results. According to one embodiment of the disclosure, the classification logic140comprises prioritization logic141and score determination logic142. The prioritization logic141may be configured to apply weighting to results provided from the RTTA logic113, the dynamic analysis logic130and/or the static analysis logic120. Thereafter, the classification logic140may route the classification results comprising the weighting and/or prioritization applied to the RTTA results, the static-based results and/or the VM-based results to the reporting logic150. The classification results may, among others, classify any malware and/or exploits detected into a family of malware and/or exploits, describe the malware and/or exploits and provide the metadata associated with any object(s) within which the malware and/or exploits were detected. Specifically, real-time traffic analysis using one or more virtual features may provide a tailored analysis such that each virtual feature may be directed to detecting characteristics typically representative of a particular exploit and/or type of attack (e.g., a ROP attack). Thus, use of the RTTA results may improve classification of a portion of network traffic (e.g., an object). The reporting logic150may generate an alert for one or more endpoint devices163and/or route the alert to a network administrator for further analysis. In addition, the reporting logic150may store the classification results (including the RTTA results, the static-based results and the VM-based results) in the storage device112for future reference. In one embodiment, results of the static analysis and/or the dynamic analysis may be provided to the RTTA logic113and/or the RTTA score determination logic114. The RTTA logic113may evaluate one or more virtual features using the static analysis and/or the dynamic analysis results. The RTTA score determination logic114may utilize the results of the RTTA logic113, the static analysis results and/or the dynamic analysis results to determine a score indicating a level of suspiciousness and/or maliciousness of the object. Additionally, the results of the evaluation performed by the RTTA logic113and/or the score determined by the RTTA score determination logic114may be provided to the classification logic140for use in classifying malware detected within the object.

When the static analysis logic120is provided at least a portion of the network traffic following a real-time traffic analysis by the RTTA logic113, the static analysis logic120may perform a static analysis on the portion of network traffic. Subsequently, the static analysis logic120may route suspicious objects (and, in many cases, even previously classified malicious objects) to the dynamic analysis logic130. In one embodiment, the dynamic analysis logic130is configured to provide, at least, an analysis of a binary included in the received network traffic and/or suspicious object(s) from the static analysis logic120.

Upon receiving at least an object from the communication interface111and/or the static analysis logic120, the dynamic analysis logic130performs processing within one or more VMs on the binary, e.g., the object is processed within the one or more VMs1311-131j(where j≥1). The processing may occur within one or more virtual machine instances (VMs), which may be provisioned with a guest image associated with a prescribed software profile. Each guest image may include a software application and/or an operating system (OS). Each guest image may further include one or more monitors, namely software components that are configured to observe and capture run-time behavior of an object under analysis during processing within the virtual machine. During the processing within the virtual machine, the object is analyzed.

FIG.2is an exemplary embodiment of a logical representation of the threat detection system ofFIG.1. The threat detection system110includes one or more processors200that are coupled to communication interface111via a first transmission medium202. The communication interface111, with the communication interface logic211located within a persistent storage210, enables communication with network devices via the Internet, the cloud computing services and one or more the endpoint devices. According to one embodiment of the disclosure, the communication interface111may be implemented as a physical interface including one or more ports for wired connectors. Additionally, or in the alternative, the communication interface logic111may be implemented with one or more radio units for supporting wireless communications with other electronic devices.

The processor(s)200is further coupled to persistent storage210via a second transmission medium203. According to one embodiment of the disclosure, persistent storage210may include (a) the static analysis logic120including a heuristics logic121, an exploit matching logic122, a vulnerability matching logic123, and a signatures database124, (b) a dynamic analysis logic130including one or more VMs1311-131j, a virtual machine manager (VMM)132, and a score determination logic133, (c) a classification logic140including a prioritization logic141, and a score determination logic142, (d) a real-time traffic analysis (RTTA) logic113, (e) a RTTA score determination logic114, and (f) a reporting logic150. Of course, when implemented as hardware, one or more of these logic units could be implemented separately from each other.

Additionally, although not shown, the RTTA logic113and/or the RTTA score determination logic114may be incorporated within any of the modules illustrated inFIG.2. For example, the RTTA logic113and the RTTA score determination logic114may be incorporated within the static analysis logic120. Furthermore, the RTTA logic113may include one or more modules. In one embodiment, the RTTA logic113includes a parsing logic, a feature detection logic (which maintains at least a representative set of observed values for one or more features), a feature-condition pairing evaluation logic (which may evaluate one or more feature-condition pairings using one or more values of a representative set of observed values, generate a result (e.g., an observation vector) and use the observation vector as an index into a truth set to determine whether one or more of the observed values satisfies the terms of the virtual feature.

Referring toFIG.3, a flowchart illustrating an exemplary method for performing a real-time traffic analysis with the threat detection system ofFIG.1is shown. Each block illustrated inFIG.3represents an operation performed in the method300of performing a real-time traffic analysis of one or more objects (or one or more sub-objects) by the real-time traffic analysis logic within a threat detection system. For convenience, in the description ofFIG.3, the term “object” will be used but the method and claims are not limited to such a scope. Referring toFIG.3, at block301, the communication interface of the threat detection system receives an object to analyze (optionally analyze on a per-sub-object basis). Subsequently, the object is passed to the real-time traffic analysis logic, which parses the object looking for features of a predefined feature set (block302).

At block303, during parsing, the RTTA logic maintains a representation of detected features of the predefined feature set as “observations.” For example, as illustrated inFIG.5, the observations (e.g., detected features of the predefined feature set) may be recorded in a table. Alternatively, the observations may be stored in any representative form, examples include but are not limited or restricted to a linked list, hash table, vector or the like. Herein, for convenience, the term “table” will be used to refer to the representative form in which the observations are stored but is not intended to limited the scope of the disclosure.

Once the object has been parsed and any values of observations have been recorded, the RTTA logic evaluates each feature-condition pairing set forth in the virtual feature against the set of all the value(s) of the corresponding observation (block304). As is defined above, the virtual feature is a combination of one or more feature-condition pairings, wherein the combination may be represented as a logical expression. Thus, each value of an observation is paired with one or more conditions. In some embodiments, depending on the content of the network traffic, an observation may have a plurality of values (e.g., when observationX, for example, is the size of an attachment to an email, observationXwill have a plurality of values when the email under analysis includes a plurality of attachments, wherein at least a first attachment has a different size than a second attachment).

Following the evaluation of each feature-condition pairing for each value of the corresponding observation, the RTTA logic generates an observation vector based on the feature-condition pairing evaluation (block305). The RTTA logic subsequently determines whether the observation vector is a member of the truth set corresponding to the virtual feature (block306). In one embodiment, the determination of whether the observation vector is a member of the truth set may be done by using the observation vector as an index into a truth table representing the truth set (e.g., whether the observation vector corresponds to a value within the truth table). When the observation vector is a member of the truth set, mark the virtual feature as “observed” (block307).

When the virtual feature is marked as observed (e.g., the observation vector is a member of the truth set corresponding to the virtual feature), a notification that the virtual feature was observed is provided to the RTTA score determination logic. When the score generated by the RTTA score determination logic, based on, inter alia, the observed virtual feature is greater than a predefined threshold, the object is provided to the static analysis logic and/or the dynamic analysis logic for further analysis (block308).

Referring toFIG.4A, a sample illustration of a virtual feature is shown. For convenience, a sample virtual feature, the virtual feature400, is shown but does not limit the scope of the claims. A virtual feature may have more or less feature-condition pairings than the virtual feature400. The feature-condition pairings as illustrated in virtual feature400are as follows: (i) FeatureA; (ii) FeatureN≥K1; and (iii) FeatureN>K2. As is illustrated, the features of virtual feature400are separated by logical operators. As discussed above, the use of one or more virtual features by a RTTA logic within a threat detection system enables the threat detection system to more efficiently and more accurately detect exploits, malware attacks and/or anomalous characteristics and/or behaviors as the virtual features provide a more tailored and directed detection. Thus, the RTTA logic more accurately determines on what data a more in-depth analysis should be performed.

Optionally, the results of the real-time traffic analysis may be provided to a classification logic to aid in classification of the object by potentially providing results of an analysis using the virtual feature that may be directed at detecting a specific exploit or type of attack.

Referring toFIG.4B, a block diagram of a sample feature set from which the virtual feature ofFIG.4Ais derived is shown. The virtual feature400is merely one embodiment of a virtual feature that may be derived from the feature set410. The feature set410includes the one or more features as seen inFIG.4B(in contrast to the feature-condition pairings as seen inFIG.4Cand mentioned above with respect toFIG.4A). As mentioned above, for convenience, the sample virtual feature400is discussed herein; however, the scope of the claims is not so limited to the virtual feature400, the feature set410, the feature-condition pairings420ofFIG.4C, or the like.

The feature set410represents the features that the real-time traffic analysis logic will attempt to detect while parsing the object. Herein, the feature set410includes the two features as illustrated boxes411and413. The feature of box411(FeatureA) is an embedded object, as illustrated in box412. As will be shown below inFIG.4C, FeatureAcorresponds to a “presence” condition such that the RTTA logic will parse the object for the presence of an embedded object. Herein, the feature is not a specific type of embedded object, but merely any embedded object (however, a feature may be directed to a specific type). The feature of box413(FeatureN) represents a numeric feature. In contrast to the FeatureAthat corresponds to the presence of a particular feature, FeatureNcorresponds to a numeric observation. Herein, the sample FeatureNcorresponds to the size of one or more embedded JavaScript instances.

Referring toFIG.4C, a sample table containing feature-condition pairings corresponding to the virtual feature400ofFIG.4Ais shown. As mentioned above, the feature-condition pairings420corresponding to the virtual feature400are set forth in the table ofFIG.4C.FIG.4C. illustrates that a feature (herein, FeatureN) may correspond to more than one condition. FeatureNis shown to correspond to the feature-condition pairings illustrated in boxes422and423(FeatureN≥K1, wherein K1=200 kilobits (kb), and FeatureN>K2, wherein K2=300 kb, respectively). As will be discussed below, the evaluation of the observations will correspond to whether at least one observed value of FeatureNsatisfies the conditions corresponding to the one or more feature-condition pairings.

Additionally, as is shown in box421, a condition may be whether a feature (herein, FeatureA) has been observed. The example feature-condition pairing corresponding to FeatureArepresents that the real-time traffic analysis logic will evaluate whether the presence of FeatureA(an embedded object) has been observed. In contrast to the numeric FeatureN, the number of embedded objects is not important, the feature-condition pairing is directed toward the presence of the feature, and thus, the number of instances of an embedded object greater than one does not change the outcome of the condition evaluation.

Referring toFIG.4D, a sample table containing the truth set corresponding to the virtual feature ofFIG.4Ais shown. As is defined above, a truth set is a set of binary vectors consisting of one or more vectors that satisfy the terms of the corresponding virtual feature. Thus, the vectors (e.g., the left-most three columns of the table430) marked with a ‘1’ in the right-most column comprise the truth set corresponding to the virtual feature400. The table430illustrates all possible vectors that may be derived from observations of the features, wherein the truth set is a subset of the table430consisting of {(0,0,1), (0, 1, 1), (1, 1, 0), (1, 1, 1)}.

Specifically, the left most column corresponds to the feature-condition pairing set forth in box421ofFIG.4C(the presence of FeatureA), the second left-most column corresponds to the feature-condition pairing set forth in box422ofFIG.4C(whether an observed instance of a value of FeatureNis greater than or equal to 2 Mb), and the third left-most column corresponds to the feature-condition pairing set forth in box423ofFIG.4C(whether an observed instance of a value of FeatureNis greater than 3 Mb). To explain further, a ‘0’ in the right-most column of table430corresponds to the feature-condition pairing not being satisfied and a ‘1’ corresponds to the satisfaction of the feature-condition pairing. As an example, the row431includes the boxes432-434, which represent one possible vector that may be derived from the evaluation of the observations of the object. Specifically, the boxes432-434represent the vector wherein none of the feature-condition pairings were satisfied and the ‘0’ in box435represents that the vector comprised of boxes432-434{(0, 0, 0)} is not a member of the truth set corresponding to the virtual feature400(e.g., failing to satisfy any of the feature-condition pairings does not satisfy the terms of the virtual feature400.

In one embodiment, the virtual feature400, the feature set illustrated in table410, the feature-condition pairings illustrated in table420and the truth set illustrated in table430are generated prior to analysis of the object. Therefore, in one embodiment, a network appliance has been preconfigured with the virtual feature400, the feature set illustrated in table410, the feature-condition pairings illustrated in table420and the truth set illustrated in table430prior to receipt of an object to analyze.

The receipt of an object for analysis by a threat detection system and the analysis of the object by, at least, the real-time traffic analysis logic within the threat detection system will be discussed below in conjunction with a discussion ofFIGS.5-7.

2. Parse Objects Looking for Features

Upon receiving network traffic (or alternatively, one or more objects, flows, group of objects, etc., may be received by the threat detection system in other manners such as via transfer from a storage device or the like), the communication interface provides at least a portion of the network traffic to the real-time traffic analysis (RTTA) logic. For convenience, the discussion herein will be directed to the analysis of an object, but the scope of the claims is not so limited.

The RTTA logic may parse the object on a per-object or per-sub-object feature basis. This enables the threat detection system to perform real-time traffic analysis at different granularities. A higher granularity may provide more detail and/or information to the static analysis logic and/or the dynamic analysis logic, when the object is passed on. However, performing RTTA at a high level of granularity (e.g., on a sub-object basis) may sometimes result in more processing for RTTA logic. A lower granularity may result in less processing for RTTA logic.

While parsing the object, the RTTA logic attempts to detect the features comprising the feature set (here, FeatureAand FeatureN). When the RTTA logic is evaluating object/sub-object for multiple virtual features, each virtual feature may use a different subset of the predefined feature set. Although not illustrated, the RTTA logic may be preconfigured with more than one virtual feature. In one embodiment, two or more virtual features may include the same subset of the feature set but the virtual features may be different (e.g., a first virtual feature may have a different order of feature-condition pairings separated by different logical operators than a second virtual feature).

3. Record Detected Features as “Observations”

Referring to a presence feature, such as FeatureAofFIG.4B, the RTTA logic maintains a representation of whether the presence feature was detected during parsing, as an “observation.” For a presence feature, once the feature has been detected, the RTTA logic does not need to adjust the record of observations (as is illustrated in row510ofFIG.5). However, with respect to a numeric feature, such as FeatureNofFIG.4B, a representation of the set of all observed values of the numeric feature is maintained, which may be recorded either immediately or upon the completion of parsing (the recordation is illustrated in row520ofFIG.5).

For example, a first feature may be a size of an embedded object and a second feature may be the number of JavaScript instances in an object. In such an example, referring to the first feature, an observation will correspond to a value for each instance of an embedded object and referring to the second feature, an observation will correspond to a value representing the number of JavaScript instances detected. Referring to a numeric feature, in one embodiment, the RTTA logic maintains the set of all observations corresponding to a feature, because a first observation may not satisfy all feature-condition pairings of the applicable feature but a second observation may satisfy one or more other feature-condition pairings of the applicable feature. Therefore, in order to evaluate the virtual feature using the entire object/sub-object, each observation (i.e., a value) of each detected instance of a numeric feature is to be recorded. Alternatively, in a second embodiment, the RTTA logic may maintain a representation of detected values corresponding to a feature that elides certain, similar values.

In one embodiment, a first numeric feature is the size of an embedded object, the RTTA logic maintains a representation of the set of all observations of embedded objects (a first value represented in the set corresponding to a first detected embedded object and a second value represented in the set corresponding to a second detected embedded object). Note that in such an example, the RTTA logic does not need to maintain duplicates (e.g., when a first embedded object and a second embedded object have the same size, a single entry in the representation of the set of all observations may be sufficient). When a second numeric feature is the number of, for example, JavaScript instances within a portion of network traffic, the RTTA maintains a representation (e.g., a count) of the number of JavaScript instances detected during parsing and records the final numeric value upon completion of the parsing.

Referring toFIG.5, a block diagram of a sample table containing recorded observation values corresponding to the virtual feature ofFIG.4Aand a sample object received by the threat detection system is shown. The values set forth as being observed in incoming traffic discussed herein, for example inFIGS.5-7, are provided merely for convenience, but the scope of the claims is not necessarily so limited.

Herein, ObservationAcorresponds to the observed value of Features and a set, ObservationN-1-ObservationN-3, corresponds to the observed values of FeatureN. According toFIG.5, the received object for analysis did not include an embedded object, hence, the value of ObservationAis “Absent,” illustrated in in row510. In contrast,FIG.5illustrates that three instances of embedded JavaScript were detected in the object, having values 160 kb, 220 kb and 310 kb, as shown in the set illustrated in row520.

As defined above, a “condition” may be either the presence of a feature or a numeric evaluation (e.g., above or below a threshold, within a specific numeric range, etc.). The evaluation of a feature-condition pairing includes determining whether the one or more detected values of an observation intersect with the subset of all possible values of that feature that satisfy the condition. The evaluation of each feature-condition pairing results in a “yes” or “no” answer. This may be represented in any manner such that there are two indicators, one representing an intersection and one representing no intersection (e.g., binary—‘1’ and ‘0’).

Referring toFIG.6, a block diagram of a sample table containing values representing feature-condition pairing evaluations corresponding to the virtual feature ofFIG.4Aand the sample object received by the threat detection system of as discussed inFIG.5is shown. As mentioned above, the values set forth as being observed in incoming traffic discussed herein are provided merely for convenience, but the scope of the claims is not necessarily so limited. Column610represents the features of the virtual feature400, column611represents the observation values as discussed with respect toFIG.5, and column612represents the result of evaluation of the feature-condition pairing using the observed values. As ObservationAhas a value of “Absent,” (i.e., an embedded object was not detected within the object), the result of the evaluation of the feature-condition pairing is negative, or ‘0’ (illustrated in row601). As the RTTA logic detected three values of FeatureN, feature-condition pairings involving FeatureNare evaluated against a set representation of those three values. Thus, illustrated in row602, two observed values of ObservationN, 220 kb and 310 kb (shown via highlighting), result in a positive evaluation of the feature-condition pairing (FeatureN>200 kb), which results in a ‘1’ in box625. Row603illustrates that only one value of ObservationN, 310 kb, results in a positive evaluation, or ‘1’ in box628(shown via highlighting in box627).

5. Generate Observation Vector

The observation vector, as defined above, is a binary representation of the evaluation of the feature-condition pairings. Specifically, the observation vector contents are ordered according to ordering of the corresponding feature-condition pairing from the virtual feature.

Referring toFIG.7, a block diagram of a sample table containing the generated observation vector corresponding to feature-condition pairing evaluations as illustrated inFIG.6is shown. As is illustrated inFIG.6, the observation of table700corresponds to the results of the feature-condition pairings set forth inFIG.5. Specifically, upon evaluating one or more values of the observations (e.g., detected features), the RTTA generates an observation vector by placing the results of the feature-condition pairings into a binary vector.

The observation vector ofFIG.6was generated by the RTTA logic by placing the result of the feature-condition pairing evaluation of FeatureA(box622) in the left-most column of table700. Similarly, the RTTA logic places the result of the feature-condition pairing evaluation of FeatureN>200 kb (box625) in the center column of table700and the result of the feature-condition pairing evaluation of FeatureN>300 kb (box628) in the right-most column of table700. Thus, the observation vector generated as a result of evaluating the feature-condition pairings using the observed values of the feature set corresponding to virtual feature400is {(0, 1, 1)}.

6. Determine Membership of Observation Vector in Truth Set

As defined above, the truth set corresponding to the virtual feature is a set of the one or more binary vectors that each satisfy the terms of the virtual feature. The determination as to whether the observation vector is a member of the truth set (and thus satisfies the terms of the virtual feature) is done by determining whether the observation vector matches a binary vector within the truth set.

Referring toFIGS.4D and7, once the RTTA logic has generated the observation vector, the RTTA logic determines whether the observation vector is a member of the truth set. As discussed above with respect toFIG.4D, the truth set corresponding to the virtual feature400consists of {(0,0,1), (0, 1, 1), (1, 1, 0), (1, 1, 1)}. The RTTA logic determines whether the observation vector is a member of the truth set (i.e., whether a vector identical to the observation vector is found among the vectors comprising the truth set); when the RTTA logic determines a match between the observation vector and a vector within the truth set exists, the RTTA logic determines the observation set is a member of the truth set. As mentioned above, in one embodiment, the determination of whether the observation vector is a member of the truth set may be done by using the observation vector as an index into the truth table.

7. Mark Virtual Feature as “Observed” and Provide Observed Virtual Feature(s) to Score Determination Logic

Once the observation has been determined to be a member of the truth set, the virtual feature is marked as “observed.” The virtual feature may be marked as observed in the preconfigured feature set, making it available to subsequent virtual features, as well as to the RTTA score determination logic. Alternatively, the RTTA may mark which virtual features, if any, were observed in a separate table and pass the table to the RTTA score determination logic.

The RTTA score determination logic determines a score for the object that indicates a level of suspiciousness for the object. The score may be based on the one or more virtual features observed. In one embodiment, the score may also be based on the value(s) of one or more observations, and/or the presence or absence of one or more values. When the RTTA score determination logic determines a score that is above a predetermined threshold, the RTTA logic passes the object, or portion of network traffic that includes the object, to the static analysis logic and/or the dynamic analysis logic, as such a score indicates at least a first level of suspiciousness and/or malicious that warrants further analysis. In contrast, when the RTTA score determination logic determines a score that is not above the predetermined threshold, the RTTA logic does not provide the object, or portion of network traffic, to the static analysis logic or the dynamic analysis logic but may instead pass the object, or portion of the network traffic, on to one or more endpoint devices.

IV. Endpoint Device Deployment

In another embodiment, the threat detection system including real-time traffic analysis logic may be deployed in an endpoint device. Similar to the deployment discussed above regarding the network appliance, the threat detection system deployed within an endpoint device includes a real-time traffic analysis (RTTA) logic and a RTTA score determination logic. Data received by the endpoint device (e.g., via a network and/or data received via alternative manners, such as through a physical connection) is passed to the RTTA logic which parses the data and attempts to detect one or more features of a predefined feature set from which one or more virtual features are derived. The endpoint device may be pre-configured to include a threat detection system including the virtual feature(s), and the feature set (e.g., the threat detection system may be pre-installed on the endpoint device or may be installed on the endpoint device at any time prior to receipt of the data to be analyzed).

As discussed above, after detecting one or more features, and recording the representation of the values of each instance of that feature, if applicable, the RTTA logic evaluates the feature-condition pairings for a virtual feature using the observed/detected features (and values) and subsequently generates an observation vector based on the evaluation. The RTTA logic determines whether the observation vector is a member of the truth set that satisfies the terms of the virtual feature and, when the observation vector is a member of the truth set, an indication that the virtual feature was observed is passed to the RTTA score determination logic. The RTTA score determination logic determines a score indicating a level of suspiciousness, maliciousness and/or anomalous characteristics and/or behaviors. When the score is above a predetermined threshold, the threat detection system performs in-depth static and/or dynamic analysis on the portion of received data in order to determine whether the portion of data is suspicious, malicious or exhibits anomalous and/or unwanted characteristics and/or behaviors.

In the foregoing description, the invention is described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.