Abstract:
An anomalous process behavior manager uses statistical information concerning running processes to detect and manage process behavioral anomalies. The anomalous process behavior manager collects per process statistical data over time, such as resource allocation statistics and user interaction statistics. Current collected statistical data is analyzed against corresponding historical statistical data to determine whether processes are behaving in expected ways relative to past performance. Appropriate corrective steps are taken when it is determined that a process is behaving anomalously. For example, the process&#39;s blocking exclusions can be revoked, the process can be uninstalled, the process and/or the computer can be scanned for malicious code, the user can be alerted and/or relevant information can be shared with other parties.

Description:
TECHNICAL FIELD 
     This invention pertains generally to computer security, and more specifically to profiling resource utilization and user interaction to detect software process behavioral anomalies. 
     BACKGROUND ART 
     In most client security applications that are installed on an end user&#39;s computer (for example, personal firewalls and antivirus software), the user and/or manufacturer of the security application defines exclusions to the blocking policies of the security products. For example, the user might define an exclusion allowing a trusted personal accounting program to transmit financial data to the Internet, or the manufacturer of anti-virus software may define an exclusion allowing its own update management program to overwrite anti-virus signatures stored on the user&#39;s hard disk. In other words, certain trusted processes are allowed to perform activities which the security software, by default, blocks processes from performing. These exclusions to blocking policies are for well known, trusted, applications. A resulting problem is that these exclusions are often wrongly carried over across upgrades, plug-in changes, and worst of all, unauthorized attachment to the trusted processes by malicious code, such as a computer virus or worm. 
     The majority of processes running on an end user computer have a standard pattern of system resource utilization and user interaction. For example, a process that has never used the network is unlikely to start using the network. Likewise, an application that uses the network heavily, but has never enumerated every file on the hard drive is unlikely to start enumerating the hard drive. 
     What is needed are methods, computer readable media and computer systems for detecting anomalous behavior by previously trusted processes, so that appropriate corrective action can be taken, such as revoking the blocking exclusions. 
     SUMMARY 
     Statistical information concerning application behavior is collected over time to determine the standard behavior of known applications. The applications are monitored as they run on a computer, and current behavioral information is collected. The current behavior is compared to the known, expected behavior, in order to determine when an application is behaving anomalously. When anomalous application behavior is detected, appropriate corrective steps are taken. 
     More specifically, an anomalous process behavior manager collects per process statistical data over time, such as resource allocation statistics and user interaction statistics. Current collected statistical data is analyzed against corresponding historical statistical data to determine whether processes are behaving in expected ways relative to past performance. Appropriate corrective steps are taken when it is determined that a process is behaving anomalously. For example, the process&#39;s blocking exclusions can be revoked, the process can be uninstalled, the process and/or the computer can be scanned for malicious code, the user can be alerted and/or relevant information can be shared with other parties. 
     The features and advantages described in this summary and in the following detailed description are not all-inclusive, and particularly, many additional features and advantages will be apparent to one of ordinary skill in the relevant art in view of the drawings, specification, and claims hereof. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram illustrating a system for utilizing resource utilization and user interaction statistics to detect and manage software process behavioral anomalies, according to some embodiments of the present invention. 
     
    
    
     The Figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein. 
     DETAILED DESCRIPTION 
       FIG. 1  illustrates a system  100  for utilizing resource utilization statistics  102  and user interaction statistics  104  to detect and manage software process  101  behavioral anomalies, according to some embodiments of the present invention. It is to be understood that although various components of the system are illustrated in  FIG. 1  as separate entities, each illustrated component represents a collection of functionalities which can be implemented as software, hardware, firmware or any combination of these. Where a component is implemented as software, it can be implemented as a standalone program, but can also be implemented in other ways, for example as part of a larger program, as a plurality of separate programs, as a kernel loadable module, as one or more device drivers or as one or more statically or dynamically linked libraries. 
     An anomalous process behavior manager  109  detects behavioral anomalies in applications  101  installed on a client computer  103  without requiring machine-external input. As noted above, processes  101  running on user computers  103  tend to have a standard pattern of system resource utilization and user interaction. By gathering per process  101  resource utilization and user interaction statistical data  102 ,  104  over time, it is possible to determine when a process  101  is behaving in an unexpected way. For example, if a process  101  that never access the network starts doing so extensively, this would indicate that perhaps the process  101  has been infected with malicious code. On a similar note, applications  101  that never prompt users to enter a lot of information (e.g., personal data) do not generally start doing so. 
     The anomalous process behavior manager  109  tracks per process system utilization statistics  102  along with user interaction statistics  104  to detect anomalous behavior. Thus, each individual application  101  is identified by its behavioral profile, as opposed to the use of static characteristics such as file name or hashes of parts of the executable. When used, for example, to identify an application  101  that is subject to security policy exclusions, this technique is superior to the use of static application  101  characteristics, because the application  101  can become compromised without these static characteristics changing. Frequently, a user will notice such behavior as “my system and/or network is slower than normal” or “application X, which has always worked before, suddenly stopped working properly or started crashing,” which the user knows based on their memory of “how it used to work.” The anomalous process behavior manager  109  takes the concept of “how it used to work” and automates it, so as to detect and manage anomalous process  101  behavior. 
     As illustrated in  FIG. 1 , the anomalous process behavior manager  101  comprises a resource utilization statistics collection module  113 , a user interaction statistics collection module  115 , a statistical information storage module  117 , an anomaly detection module  119  and an anomaly management module  121 . It is to be understood that although these modules  113 ,  115 ,  117 ,  119  and  121  are illustrated in  FIG. 1  as a separate entities, each illustrated module represents a collection of functionalities which can be implemented as software, hardware, firmware or any combination of these. The functionalities performed by these modules are described in detail below. 
     In some embodiments, the anomalous process behavior manager  109  uses three time periods. The first time period (T 1 ) is the time between statistics  102 ,  104  collections. That is to say, the resource utilization statistics collection module  113  and the user interaction statistics collection module  115  collect their respective per process  101  statistics  102 ,  104  once per T 1 . The second time period (T 2 ) is the time between attempts to detect process  101  anomalies (i.e., the anomaly detection module  119  runs every T 2 ). The third time period (T 3 ) is the history window, which is the amount of time for which statistical information  102 ,  104  is maintained. In some embodiments, resource allocations statistics  102  and user interactions statistics  104  are collected at different frequencies, or at the same frequency but at different times. It is to be understood that the specific lengths of the time periods are a variable design choice. Additionally, not all embodiments use three distinct time periods as such. 
     As will be readily apparent to those of ordinary skill in the art in light of this specification, the time periods simply describe how often statistics  102 ,  104  are gathered, how often attempts to detect anomalous process  101  behavior are made, and how much historical statistical data  102 ,  104  to maintain. The times informing the performance of these functions can vary from embodiment to embodiment, as well as from process  101  to process  101 , and even process  101  type to process  101  type. All such variations are within the scope of the present invention. 
     The resource utilization statistics collection module  113  collects per process system resource utilization statistics  102 . The implementation mechanics of this module  113  varies widely between various operating systems. For Microsoft Windows®, this module  113  can be implemented as a periodic read of the Windows Management Instrumentation (“WMI”) tables. The WMI tables contain current per process system resource utilization information, but they do not store historical information. Thus, in a Microsoft Windows® embodiment of the present invention, the resource utilization statistics collection module  113  can read the WMI tables every T 1 . In a UNIX° embodiment, the resource utilization statistics collection module  113  could be implemented similarly to the “top” application, or directly in the kernel. The specific implementation details of collecting per process resource utilization statistics  102  under various operating systems is know to those of ordinary skill in the relevant art, and their usage within the context of the present invention will be readily apparent to those of such a skill level in light of this specification. 
     Examples of per process system resource utilization statistics  102  that the resource utilization statistics module  113  collects are: a) central processing unit usage; b) disk reads (e.g., count or bytes) per T 1 ; c) disk writes (e.g., count or bytes) per T 1 ; d) total disk input/output per T 1 ; e) disk access span (e.g., breadth of disk location accesses by an application, such as accesses to a few folders, to a particular folder tree (e.g. My Documents), to a particular drive, or to a wide range of drives and folders, including separate characterizations for read locations versus write locations); f) network access/usage (e.g., open network connections, listening network “connections,” new network connections per T 1 , closed network connections per T 1 ); g) memory usage (e.g., memory allocated, virtual memory allocated); h) kernel objects (e.g., access tokens, communication devices, events, mutexes, semaphores, jobs, named pipes, process handles, sockets, threads, waitable timers); i) files opened per T 1  (calculated); j) files closed per T 1  (calculated); k) dynamic link libraries (DLLs) loaded by the process  101  (e.g., stability of the DLL working set—does the process  101  have a constant working set after initial startup, does it have a small number of primary working sets, or does the working set(s) vary?); l) processes  101  launched by the process  101 ; m) Graphics Device Interface (“GDI”) objects (e.g., windows, dialogs, child controls, etc.); and n) method of exiting (e.g., exit code or crash). Of course, the specific resource allocation statistics  102  to collect are variable design parameters, and can vary from process  101  to process  101  and embodiment to embodiment as desired. 
     The user interaction statistics collection module  115  collects per process  101  user interaction statistics  104 . As with the resource utilization statistics collection module  113 , the implementation mechanics of the interaction statistics collection module  115  varies widely between various operating systems. For example, for Microsoft Windows® the implementation could be as simple as a CBT Windows Hook, or as complicated as a kernel driver. As with collecting per process resource utilization statistics  102 , the specific implementation details of collecting user interaction statistics  104  under various operating systems is know to those of ordinary skill in the relevant art, and their usage within the context of the present invention will be readily apparent to those of such a skill level in light of this specification. 
     Examples of user interaction statistics  104  collected by the user interaction statistics collection module  115  are: a) types of user interface components displayed (such as informational, inquisitive, or high priority—informational includes windows/dialogs that do not require user input, inquisitive include windows/dialogs that require user input (yes/no dialogs, windows with input boxes, checkboxes, etc.), high priority includes windows/dialogs that are system modal or top most); b) method of notification (e.g., details about how the user interface component was brought to the user&#39;s attention—this can include things such as flashed, set to foreground, info-tip, system modal, top-most without focus, or background without focus); c) duration of notification (e.g., how long is the average user interface component from this application displayed?); d) scope of average user response (e.g., how many times did the user click the mouse or use the keyboard before this user interface component went away?); and e) time of day user interface component is displayed. The specific user interaction statistics to collect are variable design parameters. 
     The statistical information storage module  117  stores collected statistical information  102 ,  104 . The storage implementation can range from a full blown database to a neural network or PCA, to simply storing per process  101  running averages for the last detection period (T 2 ) and per process  101  running averages for the overall history window (T 3 ). Where feasible on the user&#39;s computer  103 , a database is a useful way to store and access collected statistical information  102 ,  104 . However, since the volume of information in question can be high, a database is not a feasible option on some user computer systems  103 . In such cases, simple running averages (per process  101  and optionally per process  101  type, etc.) can be used. In other embodiments, only the most recently gathered statistics  102  and  104  and per process  101  averages for T 3  are stored. In any case, the storage mechanism is typically implemented such that old data can be easily dropped, as the history window is refreshed every T 3 . 
     The specific statistical data  102 ,  104  to store is a variable design parameter. The specific implementation mechanics to use for doing so are a variable design choice. The implementation mechanics of storing resource allocation and user interaction statistics  102 ,  104  are known to those of ordinary skill in the relevant art, and their usage within the context of the present invention will be readily apparent to those of such a skill level in light of this specification. 
     The anomaly detection module  119  analyzes collected statistics  102  in order to detect process anomalies. In one embodiment, this module  119  executes every T 2 , to check if the average behavior of specific processes  101  over the most recent T 2  interval for which data exists is within T 3  standards. This could be as simple as a comparison with a difference threshold, to as complicated as passing the current values to the trained neural network or performing a time series analysis. For example, in one embodiment the anomaly detection module  119  simply compares the current average per process statistics  102 ,  104  (i.e., the running average statistics  102 ,  104  for the most recent T 2  for each process  101  being monitored) to the per process  101  average over T 3 , to determine whether the current statistics  102 ,  104  differ from the T 3  average by more than an acceptable range of variation. In other embodiments, more complicated analyses are performed, as noted above. Note that the comparison of T 2  running averages with T 3  running averages is simply one option. For example, in other embodiments current T 1  statistics can be compared to T 3  running averages. 
     Of course, what statistics  102 ,  104  to analyze and what ranges of variation are considered to be normal and/or anomalous are variable design parameters. Tighter and looser levels of variation can be classified as normal and/or anomalous for different types of gathered statistics  102 ,  104  and/or different processes  101  as desired. Additionally, what processes  101  to monitor, and whether to compare gathered statistics  102 ,  104  for a given process  101  to T 3  data for that process  101  or for multiple processes  101  (e.g., all monitored processes  101  of that type) are variable design parameters. When a process is deemed to be behaving anomalously, the anomaly detection module  119  reports the detected anomalous process activity to the anomaly management module  121 . 
     The anomaly management module  121  takes appropriate action in response to a determination that a process  101  is behaving anomalously. The specific action to take can vary as desired. For example, the anomaly management module  121  can revoke the process&#39;s  101  blocking exclusions, delete the process  101  from the computer  103 , scan the process and/or computer  103  for malicious code, notify the user and/or other processes  101 , share the information with a computer security alert network  123 , etc. 
     The implementation specifics of the anomaly management module  121  are informed by the functions it performs. For example, in embodiments in which the only response to detecting anomalous behavior is to notify the user, the anomaly management module  121  can consist of a simple notification user interface. In other embodiments, this module  119  can be implemented so as to take further action, such as scanning the process  101  (and optionally all DLLs loaded therein) for malicious code, closing the process&#39;s  101  network connections, suggesting to the user not to input any financial information to the application  101 , uninstalling the process  101 , rolling back the process  101  to an earlier version, etc. 
     Another exciting possibility is the sharing of gathered statistical data  102 ,  104  and detected anomalous process  101  behavior with a central security server  121  or through a peer to peer trusted community (not illustrated), so that other users (e.g., friends or subscribers to a network of “experts”) can benefit from the profiling activities of others (and the information can be aggregated and redistributed). 
     The implementation mechanics of taking the responsive actions described above are known to those of ordinary skill in the relevant art, and their usage within the context of the present invention will be readily apparent to those of such a skill level in light of this specification. 
     As will be understood by those familiar with the art, the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the portions, modules, agents, managers, components, functions, procedures, actions, layers, features, attributes, methodologies and other aspects are not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, divisions and/or formats. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the portions, modules, agents, managers, components, functions, procedures, actions, layers, features, attributes, methodologies and other aspects of the invention can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present invention is implemented as software, the component can be implemented as a script, as a standalone program, as part of a larger program, as a plurality of separate scripts and/or programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Furthermore, it will be readily apparent to those of ordinary skill in the relevant art that where the present invention is implemented in whole or in part in software, the software components thereof can be stored on computer readable media as computer program products. Any form of computer readable medium can be used in this context, such as magnetic or optical storage media. Additionally, software portions of the present invention can be instantiated (for example as object code or executable images) within the memory of any programmable computing device. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.