Patent Publication Number: US-7904959-B2

Title: Systems and methods for detecting and inhibiting attacks using honeypots

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation of International Patent Application No. PCT/US2006/014704, filed Apr. 18, 2006, which claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 60/672,280, filed Apr. 18, 2005, each of which is hereby incorporated by reference herein in its entirety. 
    
    
     TECHNICAL FIELD 
     The disclosed subject matter relates to the field of computer security. More particularly, the disclosed subject matter relates to systems and methods for detecting and inhibiting software attacks. 
     BACKGROUND 
     Computer viruses, worms, trojans, hackers, key recovery attacks, malicious executables, probes, etc. are a menace to users of computers connected to public computer networks (such as the Internet) and/or private networks (such as corporate computer networks). In response to these threats, many computers are protected by antivirus software and firewalls. However, these preventative measures are not always adequate. For example, there have been many instances where worms and other attacks have taken advantage of a known vulnerability in computer software (e.g., in firewall technology) the day after the public became aware of the vulnerability. Because of such a rapid launch, the patch necessary to correct the vulnerability could not be deployed in time to prevent the attack. Similar, most antivirus software relies on updates to that software so that signatures of known viruses can be utilized to recognize threats. In the case of a “zero-day” worm or virus (e.g., a worm or virus that has just been launched), most computer systems are completely vulnerable to attack because no known patch or signature update has yet been made available. 
     Due to the level of malicious activity on the Internet, organizations are deploying mechanisms for detecting and responding to new attacks or suspicious activity, sometimes referred to as intrusion prevention systems. However, many of these intrusion prevention systems are limited to protecting against already known attacks because they use rule-based intrusion detection systems. 
     Detection mechanisms, such as honeypots and anomaly detection systems, have been developed for use in more powerful reactive-defense systems. In contrast with intrusion detection systems, honeypots and anomaly detection systems offer the possibility of detecting and responding to previously unknown attacks or “zero-day” attacks. A honeypot is generally defined as a trap that is set to detect or deflect attempts at unauthorized use of information systems. However, it should be noted that honeypots do not see legitimate traffic or activity. One reason for this is because honeypots are often placed at a network location that legitimate traffic would have no reason to access. Another reason is that honeypots are usually too slow to process traffic in the time required for real world applications. Yet another reason is that honeypots do not have the capabilities to fully service the legitimate traffic because, for example, honeypots do not fully share state with the actual system. Anomaly detection systems protect a resource by monitoring attempted accesses to it and using heuristics to classify the accesses as normal or anomalous. Honeypots and anomaly detection systems offer different tradeoffs between accuracy and the scope of attacks that can be detected. 
     Honeypots can be heavily instrumented to accurately detect attacks, but they depend on an attacker attempting to exploit a vulnerability against them. This makes honeypots good for detecting scanning worms, but ineffective against manually directed attacks or topological and hit-list worms. Furthermore, honeypots are typically used only for server-type applications. 
     Anomaly detection systems can theoretically detect both types of attacks, but are usually much less accurate than honeypots. Most anomaly detection systems offer a tradeoff between false positive and false negative detection rates. For example, it is often possible to tune the system to detect more potential attacks, however, this is at an increased risk of misclassifying legitimate traffic. While it is possible to make an anomaly detection system more insensitive to attacks, this creates the risk of missing some actual attacks. Because anomaly detection systems can adversely affect legitimate traffic by, for example, dropping legitimate requests, system designers often tune the system for low false positive rates which may misclassify attacks as legitimate traffic. 
     Accordingly, it is desirable to provide systems and methods that overcome these and other deficiencies of the prior art. 
     SUMMARY 
     In some embodiments of the disclosed subject matter, systems and methods for protecting applications from attacks are provided. The received traffic (e.g., network traffic) is transmitted through an anomaly detection component. The anomaly detection component predicts whether the received traffic is a potential attack on the application and the anomaly detection component is tuned to achieve a low false negative rate. In response to receiving an indication from the anomaly detection component that the traffic may be an attack, transmitting the received traffic to a honeypot. The honeypot can be an instrumented instance of the application that shares state information with the application. It should be noted that, in some embodiments, the honeypot can be at least a portion of the protected application itself. The honeypot monitors the received traffic for attacks on the application. In response to receiving an indication from the honeypot that the received traffic is an attack on the application, the honeypot provides an instruction to repair the state information of the application. 
     In some embodiments, the repair can include rolling back the state information of the application to previously saved state information. In some embodiments, the repair can include discarding the state information of the application. In some embodiments, the repair can include initiating a new instance of the application with previously saved state information. 
     In some embodiments, a filter can be provided that filters traffic prior to transmitting the received traffic to the anomaly detection component. The filtering can be based on at least one of payload content and source of the attack. 
     In some embodiments, feedback may be provided by the honeypot. In some embodiments, feedback from the honeypot can be used to update the anomaly detection component (e.g., the predictors and models associated with the anomaly detection component) and/or the filter. In some embodiments, the feedback can be used to automatically tune the anomaly detection component and/or the filter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of an illustrative system suitable for implementation of an application that monitors traffic and protects applications from attacks in accordance with some embodiments of the disclosed subject matter. 
         FIG. 2  is a detailed example of the server and one of the clients of  FIG. 1  that may be used in accordance with some embodiments of the disclosed subject matter. 
         FIG. 3  is a schematic diagram of another illustrative arrangement suitable for monitoring traffic and protecting applications from attacks in accordance with some embodiments of the disclosed subject matter. 
         FIG. 4  is an illustrative flow diagram for protecting an application from attacks in accordance with some embodiments of the disclosed subject matter. 
         FIG. 5  is a schematic diagram of another illustrative arrangement suitable for monitoring traffic and protecting applications from attacks in accordance with some embodiments of the disclosed subject matter. 
         FIG. 6  is a schematic diagram of another illustrative arrangement suitable for monitoring traffic and protecting applications from attacks in accordance with some embodiments of the disclosed subject matter; 
         FIG. 7  is an example of a pmalloc( )-based memory allocation as used in accordance with some embodiments of the disclosed subject matter. 
         FIG. 8  is an example of a transformation of C code as performed in accordance with some embodiments of the disclosed subject matter. 
     
    
    
     DETAILED DESCRIPTION 
     In accordance with some embodiments of the disclosed subject matter, systems and methods for improved protection from attacks on an application are provided. These systems and methods provide at least one anomaly detector and at least one honeypot (sometimes referred to as a “shadow honeypot”). These shadow honeypots enable signs of an attack to be detected and enable the application to be protected from such an attack. The anomaly detector monitors all traffic routed to a protected application. Instead of allowing the application to process traffic that is considered anomalous, anomalous traffic is processed by the honeypot. As used herein, the honeypot is an instance of the protected application that shares substantially all internal state with a regular instance of the application. However, the honeypot can be provided in various approaches. For example, in some embodiments, the honeypot application can be at least a portion of the protected application itself. Using the honeypot, attacks against the honeypot are detected and the honeypot repairs the application (e.g., any incurred state changes are discarded). In other embodiments, the application can be restarted and loaded with state information that was saved prior to the attack. Legitimate traffic that is misclassified by the anomaly detector can be validated by the honeypot and can be processed correctly by the system and transparently to the user. It should be noted that a shadow honeypot can be used for protection of both server and client applications. 
     Some embodiments of the disclosed subject matter allow system designers to tune the anomaly detection system for low false negative rates, thereby minimizing the risk of misclassifying a real attack as legitimate traffic, since the honeypot removes any false positives (or a falsely triggered indication of an attack). Some embodiments can defend against attacks that are tailored against a specific site with a particular internal state. 
       FIG. 1  is a schematic diagram of an illustrative system  100  suitable for implementation of systems and methods for protecting an application from attack in accordance with some embodiments. Referring to  FIG. 1 , a system  100  that can be used in implementing some embodiments of the disclosed subject matter is shown. As illustrated, system  100  can include one or more clients  102 . Clients  102  can be local to each other or remote from each other, and are connected by one or more communications links  104  to a communications network  106  that is linked via a communications link  108  to a server  110 . Various embodiments of the present application can be implemented on at least of the server and the clients. 
     In system  100 , server  110  can be any suitable server for executing an application, such as a processor, a computer, a data processing device, or a combination of such devices. Communications network  106  can be any suitable computer network including the Internet, an intranet, a wide-area network (WAN), a local-area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), or any combination of any of the same. Communications links  104  and  108  can be any communications links suitable for communicating data between clients  102  and server  110 , such as network links, dial-up links, wireless links, hard-wired links, etc. Clients  102  can be personal computers, laptop computers, mainframe computers, dumb terminals, data displays, Internet browsers, personal digital assistants (PDAs), two-way pagers, wireless terminals, portable telephones, etc., or any combination of the same. Clients  102  and server  110  can be located at any suitable location. In one embodiment, clients  102  and server  110  can be located within an organization. Alternatively, clients  102  and server  110  can be distributed between multiple organizations. 
     The server and one of the clients, which are depicted in  FIG. 1 , are illustrated in more detail in  FIG. 2 . Referring to  FIG. 2 , client  102  can include processor  202 , display  204 , input device  206 , and memory  208 , which can be interconnected. In a preferred embodiment, memory  208  contains a storage device for storing a client program for controlling processor  202 . Memory  208  can also contain an application for protecting at least one other application from attacks. In some embodiments, applications can be resident in the memory of client  102  or server  110 . 
     Although some embodiments are described herein as being implemented on a client or server, this is only illustrative. The application can be implemented on any suitable platform (e.g., a personal computer (PC), a mainframe computer, a dumb terminal, a data display, a two-way pager, a wireless terminal, a portable telephone, a portable computer, a palmtop computer, a H/PC, an automobile PC, a laptop computer, a personal digital assistant (PDA), a combined cellular phone and PDA, etc.) to provide such features. 
     A simplified illustration of the system of some embodiments is illustrated in  FIG. 3 . As shown, system  300  includes an anomaly detection sensor  310 . Anomaly detection sensor  310  receives traffic from the network  320 . Anomaly detection sensor  310  can route traffic to either a protected application  330  or a honeypot version of the application  340 . In some embodiments, anomaly detection sensor  310  can route traffic to protected application  330  or the honeypot  340  based on a prediction made by anomaly detection sensor  310 . For example, anomaly detection sensor  310  can make a prediction on traffic from the network  320  based at least in part on performing a payload analysis or analyzing network behavior. 
     As described previously, the honeypot version of the application  340  is an instance of the protected application  330  that shares substantially all internal state with a regular instance of the application. In some embodiments, the honeypot application can be at least a portion of the protected application itself. Using honeypot  340 , attacks against the honeypot  340  are detected and any incurred state changes are discarded. Traffic  320  that is misclassified by the anomaly detection sensor  310  can be validated by the honeypot  340  and can be processed correctly by the system  300  and transparently to the user. Both the honeypot  340  and the protected application  330  are connected to the same process state information  350 . Generally, process state information  350  can include, for example, variable values, memory allocation information, state information related to associated components or devices, or any other suitable information that may be needed to define the current state of a process. 
     In some embodiments, honeypots, such as honeypot  340 , can provide anomaly detection sensors or any other suitable component of system  300  with feedback. As shown in  FIG. 3 , honeypot  340  can provide feedback  360  to anomaly detection sensor  310 . For example, honeypot  340  can provide information relating to the attack to the anomaly detection sensor  310 . Anomaly detection sensor  310  can use the feedback  360  from honeypot  340  to update the predictors used within anomaly detection sensor  310  or use the feedback  360  in any other suitable approach for tuning anomaly detection sensor  310 . 
       FIG. 4  is an illustrative flow diagram illustrating the interaction of the anomaly detection sensor and the honeypot in accordance with some embodiments. Traffic from a network is received by the system at the anomaly detection sensor ( 410  in  FIG. 4 ). The anomaly detection sensor determines whether or not the received traffic is an attack on the application ( 420  in  FIG. 4 ). 
     If the anomaly detection sensor predicts that the received traffic is an attack, the received traffic is routed to the honeypot ( 430  in  FIG. 4 ). Traffic that reaches the honeypot is processed by the honeypot and monitored for attacks ( 440  in  FIG. 4 ). If no attack is detected by the honeypot, the processing of the code is handled transparently by the system ( 450  in  FIG. 4 ). From the user&#39;s point of view, it appears as though the traffic was processed by the protected application. The honeypot can also inform the anomaly detection sensor that it made a wrong prediction ( 450  in  FIG. 4 ). The anomaly detection sensor can use this information to improve future predictions made by the anomaly detection sensor. Otherwise, if the honeypot determines that the received traffic is an attack, the honeypot can discard any state changes caused by the attack ( 460  in  FIG. 4 ). The honeypot can also inform the anomaly detection sensor of a correct prediction. Information relating to correct predictions can also be used to improve future predictions made by the anomaly detection sensor. 
     Alternatively, if the anomaly detection sensor predicts that the received traffic is not an attack, the received traffic is routed to the protected application ( 470  in  FIG. 4 ). In this case, the received traffic is not routed to the honeypot version. 
       FIG. 5  shows another illustrative arrangement for the attack detection system in accordance with some embodiments. In the arrangement of  FIG. 5 , system  500  includes a filtering engine  510 , an array of anomaly detection sensors  520 , a honeypot version of the application  530 , a protected application  540 , a current process state  550 , and a saved process statement for rollback  560 . 
     Filtering engine  510  receives traffic from the network  570 . Filtering engine  510  can be used by system  500  to block known attacks within the received traffic  570 . The filtering can be done, for example, based on payload content, source of the attack, a combination of any suitable filtering methodology, or by any other suitable approach for filtering the received traffic. As shown in  FIG. 5 , filtering engine  510  can be configured as a layer between the array of anomaly detection sensors  520  and the traffic from the network  570 . Filtering engine  510  can be used to drop specific types of traffic before any additional processing is performed. 
     If traffic from the network  570  passes through filtering engine  510 , the array of one or more anomaly detection sensors  520  can be used to process the received traffic  570 . Examples of types of anomaly detection sensors  520  that can be used in system  500  include sensors that perform payload analysis and network behavior anomaly detectors. However, it should be noted that various anomaly detection sensors can be used in various embodiments of the disclosed subject matter. In some embodiments, results can improve if the anomaly detection sensors  520  are tuned to a high sensitivity. While this increases the amount of false positives reported by the anomaly detection sensors  520 , it further ensure that attacks do not reach the protected application  540 . 
     Based at least in part on the prediction of the one or more anomaly detection sensors  520 , system  500  invokes either the regular instance of the protected application  540  or the honeypot version of the application  530 . As described previously, the honeypot version of the application  530  is an instrumented instance of the application that shares substantially all internal state with a regular instance of the application. Using honeypot  530 , attacks against the honeypot  530  are detected and any incurred state changes are discarded. Both the honeypot  530  and the protected application  540  are connected to the same process state information  550 . 
     However, in response to detecting an attack or failure, honeypot  530  can rollback any state changes to a known good state. For example, honeypot  530  can rollback to the state the protected application  540  had prior to processing the malicious request (e.g., a saved process statement for rollback  560 ). Alternatively, honeypot  530  can restart the application or initiate a new instance of the application having previously saved state information in response to detecting an attack. 
     In some embodiments, honeypot  530  can provide filtering engine  510 , one or more anomaly detection sensors  520 , or any other suitable component of system  500  with feedback. For example, in response to the honeypot  530  detecting an actual attack, determining a false positive, or detecting no attacks, honeypot  530  can provide information to one or more of the anomaly detection sensors  520 . The one or more anomaly detection sensors  520  can use the feedback  580  from honeypot  530  to update the predictors used within the one or more anomaly detection sensors  520  or use the feedback  580  in any other suitable approach for tuning the one or more anomaly detection sensors  520 . For example, the one or more anomaly detection sensors  520  can use the information to block similar attacks in the future and/or learn from past correct and incorrect predictions. In another example, predictors or prediction models can be updated if no attacks are detected. Any other suitable approach for self-tuning and updating the one or more anomaly detection sensors  520  can also be used. 
     In another example, in response to the honeypot  530  detecting an actual attack, determining a false positive, or detecting no attacks, honeypot  530  can provide information (e.g., feedback  590 ) to filtering engine  510 . Filtering engine  510  can use the feedback  590  from honeypot  530  to update the filters used by filtering engine  510  or use the feedback  590  in any other suitable approach for tuning the filtering engine  510 . For example, filtering engine  510  can use the feedback  590  to block similar attacks in the future and/or learn from past correct and incorrect predictions. Any other suitable approach for self-tuning and updating the filtering engine  510  can also be used. 
     It should be noted that traffic through system  500 , including attack traffic, can interact with the protected application through processing by filtering engine  510  and one or more anomaly detection sensors  520 , where side channel access is not permitted. 
     Some embodiments can be integrated with both servers and clients. 
     In some embodiments, the shadow honeypot can be tightly coupled with a server. The server application can be protected by having suspicious requests diverted to a honeypot version of the server application in which the server application and the honeypot have mirrored functionality and state. The protected application can be, for example, a web server. 
     In some embodiments, the shadow honeypot can be tightly coupled with a client. The client application can be protected from, for example, passive attacks where the attacker lures a user of the client to download data containing an attack (e.g., the recent buffer overflow vulnerability in Microsoft Internet Explorer&#39;s JPEG handling). The protected application can be used to protect, for example, a web browser or any other application on the client from an attack contained in a single packet, an entire flow, or a set of flows. 
     Although some embodiments have been described and illustrated in the foregoing description as tightly coupled so that attackers cannot exploit the differences between the state of the honeypot and the state of application because both share the same state, it should be noted that the present disclosure has been made only by way of example, and that numerous changes in the details of construction and combination and arrangement of processes and equipment can be made without departing from the spirit and scope of the disclosed subject matter. For example, a loosely coupled configuration in accordance with some embodiments can be provided. The honeypot can reside on a different system and would not share state with the protected application. This loosely coupled embodiment allows the honeypot to be managed by a separate entity, thereby providing the user with the ability to outsource the management of the honeypot. 
     As shown, the filtering and anomaly detection components (e.g., filtering engine  510  and one or more anomaly detection sensors  520 ) can be tightly coupled with the protected application. Alternatively, the filtering and anomaly detection components can be centralized at a natural aggregation point in the network topology, such as the firewall. 
       FIG. 6  shows a diagram of another illustrative arrangement for the attack detection system in accordance with some embodiments. As shown in the arrangement of  FIG. 6 , the filtering and anomaly detection components can be implemented with a network processor as a scalable custom load balancer and an array of snort sensors running on computers connected to the network processor board that implement all detection heuristics. For example, the selected network processor can be an IXP1200 network processor. In some embodiments, none of the detection heuristics are implemented on it as network processors are designed for simple forwarding and lack the processing capacity required for the necessary speeds and can be harder to program and debug than other processors. However, in some embodiments, detection heuristics can be implemented on the network processor. 
     The anomaly detection component provides the ability to divert potentially malicious requests to the honeypot. For example, in web servers, a reasonable definition of the attack context is an HTTP request. For this purpose, the anomaly detection component constructs a request, runs the anomaly detection heuristics, and forwards the request depending on the outcome. Since this processing can be performed at the HTTP level, a HTTP proxy-like function is provided. The anomaly detection components for the tightly coupled honeypot server can be implemented by augmenting an HTTP proxy with the ability to apply the Abstract Payload Execution detection heuristic on incoming request and route them according to its decision. 
     For the client scenario, an alternating solution based on passive monitoring can be used. In this configuration, the TCP streams of HTTP connections can be reconstructed and the HTTP protocol can be decoded to extract suspicious objects. An object can then be examined directly by applying a heuristic algorithm. If the object is deemed suspicious, a server is launched that replays the suspicious response and directs a honeypot browser to this server. 
     Payload sifting and buffer overflow detection via abstract payload execution can be used as heuristics for anomaly detection. The payload sifting algorithm derives fingerprints of rapidly spreading worms by identifying popular substrings in network traffic. Buffer overflow detection via abstract payload execution detects buffer overflow attacks by searching for sufficiently long sequences of valid instructions in network traffic. Long sequences of valid instructions can appear in non-malicious data and this is where the honeypot can be implemented. This detection mechanism is attractive because it can be applied to individual attacks and triggers detection upon encountering the first instance of an attack. This is unlike many anomaly detection mechanisms that are required to witness multiple attacks before flagging the attacks as anomalous. 
     In some embodiments, a code transformation tool can be used to create a honeypot. This tool receives the original application as input and inserts in the honeypot code. While memory-violation errors are primarily described in these embodiments, it should be noted that various embodiments can also be used to catch and detect other violations. TXL, a hybrid functional and rule based language, can be used to perform the code transformations, where GCC C (GNU C Compiler) can be used as the front end. TXL is well suited for performing source-to-source transformation, however a very wide variety of languages could also be used. The grammar responsible for parsing the source input is specified in a notation similar to Extended Backus Naur. 
     In some embodiments, the honeypot is created by moving all static buffers to the heap. Static buffers can be moved to the heap by dynamically allocating each buffer upon entering the function in which it was previously declared. These buffers can be de-allocated upon exiting the function, whether implicitly (e.g., by reaching the end of the function body) or explicitly (e.g., through a return statement). It should be noted that because the allocated memory regions are instrumented, illegal accesses regarding pointer aliasing are detected. 
     For memory allocation, a new version of malloc( ), called pmalloc( ), was created.  FIG. 7  is an illustrative example of a pmalloc( )-based memory allocation in accordance with some embodiments. As shown in  FIG. 7 , the pmalloc( )-based memory allocation allocates three memory pages  710 . Besides the basic one page, pmalloc( ) allocates two additional zero filled, write-protected pages  720  that bracket the requested buffer. The extra pages can be mmap( )&#39;ed from /dev/zero as read-only. Because mmap( ) operates at memory page granularity, memory requests are rounded up to the nearest page. The pointer  730  that is returned by pmalloc( ) can be adjusted to immediately catch any buffer overflow or underflow. 
       FIG. 8  shows an example of a code translation in accordance with some embodiments. The original code is shown in block  810  and the translated code is shown in block  820 . Buffers that are allocated via malloc( ) can be switched to pmalloc( )  830 . This can be achieved by examining declarations in the source and transforming them to pointers where the size is allocated with a malloc( ) function call. The C grammar can also be adjusted to free the memory  840  before the function returns. 
     Any overflow or underflow on a buffer allocated via pmalloc( ) can cause the process to receive a Segmentation Violation (SEGV) signal, which is caught by a signal handler that is added to the source code in main( ). The signal handler notifies the operating system to abort all state changes made by the process while processing this request. To do this, a new system call to the operating system can be added called transaction( ). The transaction( ) function is conditionally invoked at three location in the code at the discretion of the shadow_enable( ) function. The shadow_enable( ) function returns true when the honeypot version of the code is running. 
     The first location that transaction( ) is placed is inside the main processing loop prior to beginning the handling of a new request. This indicates to the operating system that a new transaction has begun. The operating system makes a backup of all memory page permissions and marks all heap memory pages as read-only. As the process executes and modifies these pages, the operating system maintains a copy of the original page and allocates a new page, which is given the permissions the original page from the backup, for the process to use. This is done in the same way that copy-on-write works in modern operating systems. This call to transaction( ) can be placed manually by, for example, the programmer or system designer. Both copies of the page are maintained until transaction( ) is called again, as will be described below. 
     After the end of handling a request, inside the main processing loop, is the second location transaction( ) can be invoked. It is done here to indicate to the operating system that a transaction has successfully completed. The operating system then discards all original copies of memory pages that have been modified during processing the request. This call to transaction can also be placed manually. 
     The third location that transaction can be invoked is inside the signal handler. This location can be installed automatically by some embodiments. This is done to indicate to the operating system that an attack (exception) has been detected. The operating system then discards all modified memory pages by restoring the original pages. 
     In some embodiments, a similar mechanism as the one described above can be built around a file system. This can be done, for example, by using a private copy of a buffer cache for the processing done in the honeypot version of the code. 
     Although the invention has been described and illustrated in the foregoing illustrative embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the invention can be made without departing from the spirit and scope of the invention, which is limited only by the claims that follow. Features of the disclosed embodiments can be combined and rearranged in various ways within the scope and spirit of the invention.