System and method for active data collection in a network security system

A network security system comprises a plurality of sensors, a management server, and a data collection module. The plurality of sensors receive first data associated with potential attacks on the system. The manager server is coupled to at least one sensor and correlates at least a portion of the first data to detect potential attacks on the system. The data collection module is coupled to the manager server and generates at least one request for second data based upon at least one of the first data and the correlated data. The data collection module communicates the request to at least one source different from the plurality of sensors.

TECHNICAL FIELD OF THE INVENTION

This invention relates to intrusion detection systems and, more specifically, to a system for active data collection in a network security system.

BACKGROUND OF THE INVENTION

Intrusion detection systems are used by an enterprise to detect and identify unauthorized or unwanted use (commonly called an attack) of the enterprise's computer network, which normally comprises a large number of nodes and network operations centers. In general, these enterprise intrusion detection systems receive data using sensors or other intrusion detection devices. The system then scans the incoming data for specific patterns in network traffic, audit trails, and other data sources to detect malicious activity. The data that is received in the input stream is passively gathered. In other words, traditional intrusion detection systems are reactive in the sense that they wait for data to be sent to it before performing any sort of correlation or other data processing. As a result, certain additional data that may be useful in detecting malicious activity may not be considered by the system.

SUMMARY OF THE INVENTION

In accordance with the present invention, the disadvantages and problems associated with traditional enterprise intrusion detection systems have been substantially reduced or eliminated.

In one embodiment of the present invention, a network security system comprises a plurality of sensors, a management server, and a data collection module. The plurality of sensors receive first data associated with potential attacks on the system. The management server is coupled to at least one sensor and correlates at least a portion of the first data to detect potential attacks on the system. The data collection module is coupled to the manager server and generates at least one request for second data based upon at least one of the first data and the correlated data. The data collection module communicates the request to at least one source different from the plurality of sensors.

In another embodiment of the present invention, a method for providing network security comprises receiving first data at a sensor, the first data associated with potential attacks on the system. The method continues by correlating at least a portion of the first data to detect potential attacks on the system. The method continues by generating at least one request for second data based upon at least one of the first data and the correlated data. The method concludes by communicating the request to at least one source different from the sensor.

The invention has several important technical advantages. Various embodiments of the invention may have none, some, or all of these advantages. One advantage of the present invention is that it performs active data collection and correlation using a data collection module and data sources. By using active data collection techniques, the system considers specific data that is useful in detecting malicious activity but that was not received by sensors in the normal course of operation. As a result, the system can perform enhanced correlations of data that may not otherwise have been performed. The system is therefore better equipped to detect and resolve attacks on the enterprise.

The data collection module is disconnected from the normal event flow and the correlated event flow. As a result, the active data collection activities of the system do not impair the normal processing of data by sensors, manager servers, or global manager servers.

Other technical advantages of the present invention will be readily apparent to one skilled in the art from the description and the appended claims.

FIG. 1illustrates an intrusion detection system10distributed across an enterprise system according to one embodiment of the present invention. Intrusion detection system10comprises a plurality of sensors20, one or more manager servers30, global server40, and console50. System10also comprises data collection module60and data sources70. These elements of system10may be communicatively coupled using an internal network80. In general, system10performs active data collection and correlation using data collection module60and data sources70. By using active data collection techniques, system10considers specific data that is useful in detecting malicious activity but that was not received by sensors20in the normal course of operation. As a result, system10can perform enhanced correlations of data that may not otherwise have been performed. System10is therefore better equipped to detect and resolve attacks on the enterprise. The additional data can be actively requested from public or private data sources70that are external and/or internal to system10.

The “enterprise” may comprise any business, government, organization, or other entity that has multiple network channels or ports to a network100. Network100may include any suitable portions of an external network and/or an internal network. In addition, the business enterprise may further include links to portions of an internal network. In this regard, intrusion detection system10monitors network communications on internal links as well. For example, a business enterprise may include three ports for external network communications including email, internet, and dialup. In this example, intrusion detection system10monitors network communications on the three external ports. Based upon data received in these input streams and data actively collected by module60, system10attempts to detect, locate, or block an attack on the business. An “attack” may be any malicious, destructive, or suspicious activity communicated from a source external and/or internal to the portion of the enterprise protected by system10. Attacks may include viruses, Trojan horses, worms, or any other piece of code or data that represents at least a portion of an unwanted attempt to access the protected portion of the enterprise.

Internal network80may include one or more intranets, local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), or any other suitable enterprise network. Network80may, for example, communicate Internet Protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, and/or other suitable messages between network addresses. According to particular embodiments, messages between the levels may be in one or more formats including Intrusion Detection Message Exchange Format (IDMEF), binary format, and/or any other appropriate format.

Network100represents any network not protected by intrusion detection system10. Accordingly, network100communicably couples system10with other computer systems. Network100may, for example, communicate Internet Protocol (EP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, and/or other suitable information between network addresses. Network100may include one or more intranets, local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of the Internet, and/or any other communication system or systems at one or more locations. An external client system (not shown) may be any computer, enterprise or non-enterprise, which is trying to access the portion of internal network80protected by intrusion detection system10. As used in this document, the term “computer” is intended to encompass a personal computer, server, mainframe, terminal, workstation, network computer, kiosk, wireless data port, wireless telephone, personal digital assistant (PDA), one or more processors within these or other devices, or any other suitable processing device.

Each sensor20is located at a network port that receives TCP/IP packets or other similar network communications from network100. Generally, sensor20processes the raw data to detect the presence of an attack and outputs at least the raw data and, when appropriate, a corresponding alert message. In certain embodiments, sensor20may also generate messages, such as a packet flow, based on the raw data received from network100. All of the communications between one or more sensors20and one or more manager servers30may be referred to as an event flow.

Sensor20may use any suitable detection technique to process incoming data and output the appropriate messages. For example, sensor20may use algorithms, signatures, scripts, or any suitable detection or comparison technique to process packet headers, packet payloads, and/or any other incoming data. Sensor20could include any suitable combination of hardware, software, or firmware to receive data from sources via network100, process the raw network data, and communicate results to higher levels. For example, sensor20may comprise a computer, server, lower-level intrusion detection system, firewall, or any module written in any appropriate computer language such as, for example, C, C++, Java, Perl, and others. It will be understood that while sensor20is illustrated as a single multi-tasked module, the features and functionality performed by this engine may be performed by multiple modules such as for example, a sensor module and a packet flow generation module. Additionally, to help ensure that each port is properly monitored, each sensor20may be associated with a redundant slave sensor which is operable to assume substantially all of the functionality of the sensor20in the event of any failure of sensor20.

Manager server30represents any hardware or software module that controls or monitors one or more servant nodes, such as a sensor20. In one example, each manager server30includes a correlation engine110and a correlation ruleset120for receiving and correlating data from sensors20and/or data collection module60. Generally, through correlating and aggregating lower-level communications, manager server30is capable of detecting an attack occurring within the enterprise and dynamically responding to such a threat.

According to certain embodiments, manager server30comprises a general-purpose personal computer (PC), a Macintosh, a workstation, a Unix-based computer, a server computer, or any other suitable processing device. Additionally, to make system10more robust, each manager server30may be associated with a redundant manager server which is operable to assume substantially all of the functionality of manager server30in the event of a failure of the associated manager server30. AlthoughFIG. 1provides one example of a server30that may be used with the invention, system10can be implemented using computers other than servers, as well as a server pool. Manager server30may include any hardware, software, firmware, or combination thereof operable to receive communications from lower levels, appropriately process the communications, and dynamically respond.

Global server40may comprise a general-purpose personal computer (PC), a workstation, a Unix-based computer, a server computer, or any other suitable processing device. AlthoughFIG. 1provides one example of global server40that may be used with the invention, system10can be implemented using computers other than servers, as well as a server pool. Global server40may include any hardware, software, firmware, or combination thereof operable to process, control, and monitor system10at the highest logical level.

Global server40may include an archive database130to store the raw archival data collected from various nodes of system10for later processing, retrieval, or searches. Archive database130may include any memory or database module and may take the form of volatile or non-volatile memory comprising, without limitation, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), removable media, or any other suitable local or remote memory component. In one embodiment, archive database130comprises stored IP information such as, for example, TCPDump data corresponding to each packet flow. Archive database130may include any other data such as, for example, historical operator commands or response messages previously communicated to lower nodes. Global server40may include a global correlation engine140that is further operable to process information in archive database130to detect the presence of a substantially long-term or multi-staged attack that had previously gone undetected by sensors20and manager servers30. AlthoughFIG. 1illustrates archive database130as residing internally to global server40, archive database130may reside externally or at any other location or locations accessible by global server40or other components of system10.

Global server40may further comprise a filtering ruleset150. Filtering ruleset150allows system10to reduce the analyst's workload through filtering the information communicated to console50. Ruleset150comprises instructions, algorithms, or any other directive which largely controls the information communicated to console50. AlthoughFIG. 1illustrates filtering ruleset150as residing internally to global server40, filtering ruleset150may reside externally at one or more manager servers30or at console50without departing from the scope of this disclosure.

Console50may represent any computer that may comprise input devices, output devices, mass storage media, processors, memory, or other components for receiving, processing, storing, and/or communicating information. Intrusion detection system10may communicate the intrusion information, such as correlated and/or filtered data, to console50so that a user, such as a supervisor or administrator, may view and process the intrusion information. It will be understood that there may be any number of consoles50used in system10such as, for example, one or more operator or analyst consoles50and a supervisor console50. Console50may include a graphical user interface (GUI)52that tailors and filters the data presented to the user. Generally, GUI52provides the user of console50with an efficient and user-friendly presentation of data and events occurring in system10.

Data collection module60comprises any suitable combination of hardware and software to perform active data collection and correlation techniques. Data collection module60requests and receives information from data sources70using one or more requests170. The information is returned in the form of one or more responses172. Each request170may be based upon a kernel of information already received within system10, such as, for example, data associated with one or more attack events on system10, data correlated by servers30and/or40, or even one or more responses172. Data sources70comprise sources of data that are either internal or external to system10. For example, a data source70may comprise a geographic location server that translates between IP addresses and geographic locations. A data source70may also comprise an internet directory server, such as a Domain Name System (DNS) server, that translates between domain names and IP addresses. A data source70may also comprise a security vulnerability device, such as a scanner (e.g., Nessus), a server, or a database. Such a device may be used to launch covert gathering agents directly at an attacker.

FIG. 2illustrates an example operation of system10. In particular, sensors20communicate data to manager servers30in an event flow200. Event flow200represents information about possible attacks such as raw data, alert messages and/or logs that are passively received in the normal course of operation. Correlation engine110of one or more manager servers30correlates the information communicated in the event flow200according to a correlation ruleset120and communicates a correlated event flow202to operator console50, such as via global server40. This sort of correlation therefore depends on and is limited to information provided to manager servers30by an event input stream that is passively received by sensors20. From time to time, it may be advantageous to collect and correlate additional information about a particular event in order to determine potential malicious activity. For example, it may be useful to discover the geographical location of an attack in order to determine whether it is related to another attack. Or, it may be useful to determine if a particular attacker is using a particular operating system to better understand the nature of the attack and the attacker. In these instances, manager servers30may launch a data collection session by communicating a query204to data collection module60. The query204may include a kernel of information that may be used by data collection module60to generate an information request170. Characteristics from the kernel of information are used to request more information from data sources70. Data collection module60is disconnected from the normal event flow200and the correlated event flow202. As a result, the data collection activities of module60do not impair the normal processing of data by sensors20, and servers30and40.

The kernel of information used to request more information from data sources70may be a piece of raw data received by a manager server30, such as, for example, the source IP address of an attacker; the time of day of a particular attack; or the port on which a sensor20received the raw data. The kernel of information may also be the result of a correlation of data performed by a server30, such as, for example, the common source IP address of a series of attacks; the range of times of a series of attacks; or the common port on which a series of attacks were initiated.

An example is used to illustrate the process of generating requests170and receiving responses172. It is generally difficult to correlate attacks made from open wireless networks. Users may connect to open wireless networks from the safety of a roadside vehicle and attack or probe a remote site. After attacking, they can simply drive to another open network and attack with a different tool or probe to that network to gain more information about it. The explosion of WI-FI technology in home and office use makes it relatively easy to find and penetrate these networks as “free” connections to the internet. This attack method allows the attacker to gather data on a remote network very quickly in seemingly disconnected intrusion attempts. Since the attacks come from different network addresses and many times from different service providers at different times, there is no clear correlation to tie them together other than geographic location.

After receiving one of these attacks, manager server30may send a query204to data collection module60to collect and correlate additional information about one or more of the attacks. For example, data collection module60may try to discover the geographical location of the attacks. A “trace route” to the attacking FP address and a DNS query on the attacking IP address may result in the determination that the attack came from Malibu, Calif., or nearby. In particular, data collection module60may generate a request170of a geographic location server using one or more known IP addresses in order to determine the geographic location of the attack. The geographic location server may indicate the geographic location information as one or more responses172to data collection module60. Another request170may be initiated to determine whether other attacks have been made in this area in the last several days. No results may be found.

In an hour, another attack may be detected and another query204may be sent to the data collection module60. The geographic location of this attack may also be determined to be Malibu, Calif., even though the service provider and the IP address are in every other way unrelated to the first attempt. This time, when another request170is made to identify other attacks from this geographical area, the first attack is found, and the two are correlated by data collection module60. This additional correlation information206is communicated to console50, such as in the form of a relationship score. This relationship score indicates the likelihood that the two attacks are related, based on previous experiences or possibly through the use of a Bayesian network weight.

In time, as the attacker uses different networks and attacks, these attacks are added to the group of attacks from Malibu, Calif. Data collection module60may correlate information regarding each of these attacks. When a threshold number of such attacks has been identified, this information may also be brought to the attention of the operator in the form of correlation information206, and the intrusion attempts are plotted on a map with times of attack noted. The operator at console50may recognize a pattern and contact local authorities in Malibu, Calif. with information that could lead to the arrest and conviction of the attacker.

Although in this case, the geographical location is the information that is actively gathered, there are several other types of information that could be requested by data collection module60. For instance, a vulnerability scan of the attacking machine can reveal a vulnerability signature that can be correlated with other data or used to “hack back” the attacking machine. Logs requested from SQL, WEB, MAIL, and firewall servers could give an indication of other activities the attacker has been involved in. It may be that the volume of these logs is too large to be included during normal operation but can be requested under special circumstances. In the case that the attacker is directly connected to the network being attacked (for instance in the case of an “insider” attack), exhaustive information and packet logs can be requested of the switch connecting that user to the network. The number of items of data that can be requested or obtained are significant.

After responses172are received, data collection module60may perform correlation with other events, such as events stored in one or more archive databases130, that may not up to that point have been associated with the original event. For example, after requesting DNS records from a DNS server, it may be determined that event A and event B are from source IP addresses under the same administrative control of a single company. They may be loosely or tightly associated as a result of this new information. In addition, this connection may prompt further information gathering (e.g., for instance an exhaustive list of IP addresses under the same administrative control). On the other hand, additional information may serve to weaken previous correlative work. For example, since two attacks occurred together in time it would suggest that they may be related. However, upon doing a vulnerability scan of the machines, it may become clear that one is being controlled through a well known “drone” program while the other is an attack through a dialup network commonly used for attacks, and no evidence of a similar “drone” program exists.

By actively requesting this information and considering it in further correlation, data collection module60increases considerably the number of attacks found and the amount of understanding of the current state of the network being monitored.

FIG. 3illustrates an example embodiment of data collection module60that includes event state module250, data collection engine252, and correlation engine254. Event state module250, data collection engine252, and correlation engine252comprise any suitable combination of hardware, software, and firmware to perform the described functionality. Although these elements of data collection module60are illustrated as separate components, it should be understood that one or more of these elements may be integrated with other elements of data collection module60or with other elements of system10, such as sensors20, manager servers30, or global manager servers40.

In general, event state module250is responsible for maintaining state information for events that are the subject of queries204. Examples of state information maintained by event state module250include additional data that has been requested by data collection engine252; additional data that has been received in response to a request170; additional data that is yet to be requested; and correlated data. Data collection engine252is responsible for performing the active data collection process, including communicating requests170and receiving responses172. Data collection engine252may initiate several data collection processes in response to one or more queries204, or in response to one or more previous responses172. Event state module250maintains state information to keep track of the various data collection processes that have been or will be initiated and/or completed. Correlation engine254is responsible for correlating the additional data received in a response172with other data, such as data stored in archive database130, data stored in event state module250, other data correlated by correlation engine254, other data correlated by other correlation engines110, or any other data within system10.

In operation, when information about a new event or a group of events is received, in conjunction with a query204, it is recorded in the event state module250. Module250keeps track of the state of all of the current events for which additional information is being gathered. After the state information is recorded, an appropriate data collection process is invoked by data collection engine252. For example, a DNS query process may be invoked in order to generate an appropriate request170from a DNS server data source70. The DNS query process would be responsible for obtaining the information and returning that information to the state module250so that the new state can be computed. It may be that the information cannot be obtained or that the data collection engine252fails to report back. In this case, an automated timer can be implemented to recalculate a state of old requests170and decide what to do. For example, after a timeout, a previous request170may be reissued a predetermined number of times. Alternatively, or in addition, the previous request170may be discarded after a certain number of attempts have failed.

Data collection engine252may receive responses172that may be communicated to event state module250. When the state information is updated with the new information gathered, the information is handed to the correlation engine254which operates similarly to the correlation engine110of manager service30. Correlation engine254correlates data collected by data collection engine252with other data collected by engine252, data received by sensors20, or with the results of other data correlation to produce results. If a good decision can be made, the new correlated data206is sent to the appropriate console50. On the other hand, additional information may yet be required, in which case the incident would be again given to the event state module250and additional information requested by data collection engine252.

Another example may be used to illustrate the operation of data collection module60. System10receives many attacks upon a specific host from a variety of sources. A query204is sent to the data collection module60to try to find a correlation between these attacks. The correlation engine254of the data collection module60first tries to determine if the geographic source of the attacks is similar. If this fails, the correlation engine254tries to determine if the DNS administrative contact is the same. If this fails, the correlation engine254queries the event history for time correlation between each pair of source IP addresses to see if they have acted in concert before. This type of querying can involve one or more requests170and responses172to collect additional information from data sources70that may be useful to correlation engine254. The querying can go on until some type of correlation between the attacks is found. In that case, an appropriate action can be taken. Alternatively, the querying may go on for a period of time without leading to any correlation, at which point it may be terminated. In either case, the querying may be recorded in the event state module250.

In another example, system10receives an attack from a host. The data collection module60determines using a local lookup that this host has previously had a “counter agent” installed on it by system10to monitor activity. An information request170is put into the queue for a list of recent network connections on that host. When the host reports in, as it does periodically, that information is given to the event state module250. Each entry in the list is correlated with previous attackers by correlation engine254, and it is found that more than two hosts that have attacked system10have had connection to this third IP address. An incident that tracks these together is created to gather evidence of this third party using intermediate machines to attack system10. A scan is sent to the “real” attacker and its vulnerability is assessed. Appropriate action, such as a counter attack or communicating with an internet service provider can now be taken.

Therefore, there are a great variety of times when by requesting additional information, much smarter decisions of what to do can be made by correlation engines within system10. Data collection module60may be used to enable this type of active information gathering.