Blocking unidentified encrypted communication sessions

Techniques are described for blocking unidentified encrypted communication sessions. In one embodiment, a device includes an interface to receive a packet, an application identification module to attempt to identify an application associated with the packet, an encryption detection module to determine whether the packet is encrypted when the application identification module is unable to identify an application associated with the packet, and an attack detection module to determine whether the packet is associated with a network attack, to forward the packet when the packet is not associated with a network attack, and to take a response when the packet is associated with a network attack, wherein the encryption detection module sends a message to the attack detection module that indicates whether the packet is encrypted, wherein when the message indicates that packet is encrypted, the attack detection module determines that the packet is associated with a network attack.

TECHNICAL FIELD

This invention relates to computer networks and, more particularly, to network intrusion detection and prevention devices and systems.

BACKGROUND

A computer network typically includes a collection of interconnected computing devices that exchange data and share resources. These devices include, for example, web servers, database servers, file servers, routers, printers, end-user computers and other devices. Each of the variety of devices executes a myriad of different services and communication protocols. Each of the different services and communication protocols exposes the network to different security vulnerabilities.

In certain contexts, such as within a corporate enterprise network, a system administrator or other system owner would like to restrict or limit access to and usage of network resources. For example, the system administrator may wish to prevent network access by an unknown application because the unknown application may be malware, such as a virus, trojan, or other malicious software. The system administrator may also wish to prevent users from using certain applications due to various considerations. For example, the system administrator may wish to block network access to applications such as BitTorrent, Skype, or other “grayware” applications due to consumption of network resources or security concerns with respect to the applications. Therefore, the system administrator may configure a firewall or other network device to block certain applications and protocols.

In order to identify network communications associated with unwanted applications, IDSs or other security systems may analyze packet streams of the network communications and employ behavioral analysis. For example, a security system may perform deep packet inspection and apply patterns to the payloads of the packets in an attempt to identify the source software application. However, developers of applications that are commonly blocked, such as Skype, BitTorrent, and malware such as viruses and trojans, will often employ techniques to circumvent a system administrator's or other user's attempt to block these applications. For example, a virus programmer may code the virus to encrypt all transmissions in accordance with an encryption protocol. For example, the developers may code the application to encrypt an entire payload of a TCP/IP packet, including areas of the packet that would typically be unencrypted. As a result, intrusion detection system (IDS) or other security devices are unable to perform deep packet inspection and often have difficulty identifying the particular type of software application that originated the network communication.

As one example, the Back Orifice protocol depends upon a random number generator that is seeded, i.e. initialized, with a secret key. Both parties to a Back Orifice communication session know the secret key in advance of the communication session, and both parties have the same random number generator. By seeding the random number generator with the same secret key, both parties are able to encrypt data without exchanging a key during the communication session. In this way, the parties seek to avoid detection of any key exchange and session establishment. Moreover, parties to the Back Orifice protocol encrypt all data in the packet, including the Back Orifice application-layer header. Back Orifice uses a 17-byte application-layer header that is encrypted, in addition to the payload, by one party and decrypted by the other party to a communication session.

As another example, various BitTorrent client applications or other peer-to-peer client software applications implement similar encryption schemes in an attempt to avoid detection by security devices. For example, some BitTorrent software clients implement protocol encryption, message stream encryption, or protocol header encryption. Some BitTorrent clients provide an option for users to select between encrypting only the protocol header or the entire packet. To provide encryption, specifically to determine a key for encryption, some BitTorrent clients utilize a distributed hash table (DHT). A DHT includes an infohash, which is the result of a hash function performed on a file. The infohash may be used by each peer of a peer-to-peer file exchange using the BitTorrent protocol such that each peer may mutually generate an encryption key, such as an RC4 encryption key. In this manner, each peer may generate the same encryption key without exchanging the encryption key.

Enterprises may wish to block such applications for a variety of reasons. For example, a system administrator may wish to block the Back Orifice application because a malicious user may intentionally or inadvertently use Back Orifice to install a virus or to take control of a computer or server remotely. As another example, the administrator may wish to block applications that utilize a large amount of bandwidth, such as BitTorrent clients or Skype.

Conventional techniques for identifying software applications associated with encrypted data streams are problematic. For example, a conventional IDS may, for example, attempt to apply behavior analysis to the overall communication session, such as by determining an average size and frequency of data transmission for a certain port or session. If the average size and frequency of data transmission matches known characteristics for a malicious or unwanted application, the IDS may block further communication of that session. However, this method of profiling encrypting communication sessions in an attempt to identify the particular originating software application requires a long series of packet exchanges before detection of the unwanted application. Moreover, this method may fail to identify certain unwanted applications and may trigger false positives for desirable applications.

SUMMARY

In general, techniques are described for discovering and preventing network access or network utilization by software applications that employ encrypted communication. The techniques described herein may be employed by an intrusion detection system (IDS), an intrusion detection and prevention (IDP) device or other security device. An IDP may, in accordance with the techniques herein, monitor network traffic into and out of an enterprise network. The IDP determines whether a communication session is occurring in accordance with a known network protocol. For example, the IDP may identify the application in use for the communication session or the IDP may determine whether the communication session is following a known network protocol.

When the IDP cannot determine the network protocol or the application in use for the communication session, the IDP determines whether packets being exchanged as part of the communication session are encrypted. For example, in one embodiment, the IDP analyzes the randomness of the packet's TCP/IP payload. Where a packet's TCP/IP payload has a high degree of randomness, the IDP determines that the payload is likely encrypted. When the IDP determines that the payload is encrypted, and not as part of an identified encryption protocol, the IDP determines that the communication session is unwanted or malicious. Accordingly, the IDP terminates the communication session and records various elements of the communication session in an activity log.

In one embodiment, a method includes receiving a network packet, using an application-layer header of the packet, determining whether the packet is associated with an identifiable network application, when the packet is not determined to be associated with an identifiable network application, determining whether data in the packet is encrypted by calculating a randomness value of the packet from a payload of the packet that includes the application-layer header and an application-layer payload and determining that the packet is encrypted when the randomness value exceeds a randomness threshold, and, when the data in the packet is determined to be encrypted, executing a programmed response.

In another embodiment, a device includes an interface to receive a packet, an application identification module to attempt to identify an application associated with the packet, an encryption detection module to determine whether the packet is encrypted when the application identification module is unable to identify an application associated with the packet, and an attack detection module to determine whether the packet is associated with a network attack, to forward the packet when the packet is not associated with a network attack, and to take a response when the packet is associated with a network attack, wherein the encryption detection module sends a message to the attack detection module that indicates whether the packet is encrypted, wherein when the message indicates that packet is encrypted, the attack detection module determines that the packet is associated with a network attack.

In another embodiment, a computer-readable medium stores instructions. The computer-readable medium may be a computer-readable storage medium. The instructions cause a programmable processor to receive a network packet, determine whether the packet is associated with a network application using an application-layer header of the packet, determine whether data in the packet is encrypted, when the packet is not determined to be associated with an identifiable network application, by calculating a randomness value of the packet from a payload of the packet that includes the application-layer header and an application-layer payload and determining that the packet is encrypted when the randomness value exceeds a randomness threshold, and, when the data in the packet is determined to be encrypted, execute a response, and, when the data in the packet is determined not to be encrypted, forwarding the packet.

The techniques described herein may provide several advantages. For example, the techniques described herein may permit improved discovery and prevention of undesirable applications participating in a network communication session. These techniques may further permit early discovery of an undesirable application as of a first packet is transmitted by the application after a communication session is established.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating an exemplary system2in which enterprise computer network4includes intrusion detection system (IDS)10that intercepts packet flows associated with communication sessions and processes the packet flows to filter unwanted or malicious packet flows in accordance with the principles of the invention. In the example embodiment ofFIG. 1, IDS10is a single network device. Network4also includes a private enterprise computing network5that is coupled to public network6, such as the Internet. Public network6may include, for example, one or more client computing devices. Firewall9protects enterprise network5and, in particular, internal computing nodes8A-8N. Computing nodes8A-8N (computing nodes8) represent any private computing device within enterprise network5, including workstations, file servers, print servers, database servers, printers and other devices.

In one embodiment, IDS10attempts to identify applications and protocols for each communication session between computing nodes8and other computing devices in public network6. Exemplary techniques for identifying specific applications and protocols are described in greater detail in U.S. patent application Ser. No. 11/835,923, Burns et al., “Identifying Applications for Intrusion Detection Systems,” filed Aug. 8, 2007, assigned to the assignee of the present application, which is hereby incorporated by reference in its entirety.

Although IDS10may identify many of the applications and protocols in use for these communication sessions, certain applications, such as certain applications executing on computing nodes8, may attempt to hide their identities from IDS10. For example, these applications may encrypt application-layer data in the packets' payloads, such as portions of an application-layer header and/or an application-layer payload. These software applications may also begin communicating using encrypted packet streams without first performing a key exchange when establishing the communication session. The techniques described herein recognize that most legitimate applications that utilize encryption often first perform an identifiable key exchange when establishing a communication session and before beginning encryption of packets associated with that session. The techniques also recognize that illegitimate applications, on the other hand, attempt to hide the key exchange by performing a key exchange during a different communication session, through the use of a hash function performed on the communicated data, by agreeing on a key before initiating the communication session, by exchanging the key in an out-of-band communication, or through some other nonstandard method.

Moreover, the techniques further recognize that many software applications, in an attempt to avoid detection, frequently encrypt all or portions of an application-layer header within each packet as well as application-layer data carried by the packet. For example, these programs may encrypt the entire payload of an Internet protocol suite packet (TCP/IP) packet. The techniques described herein recognize that, generally, a legitimate application following a well-known encryption protocol, such as the secure socket layer (SSL) protocol does not encrypt the application-layer header, the transportation communication protocol (TCP) header, or the Internet protocol (IP) header, but tend to only encrypt application-layer data within the packet. It has been determined that applications attempting to avoid detection may, however, encrypt the application-layer header, as well as the application-layer payload, without encrypting the TCP header or the IP header. Packets encrypted by these applications are generally referred to herein as “fully encrypted,” because both the application-layer header and the application-layer payload are encrypted in these packets. IDS10attempts to identify those communication sessions in which the packets are fully encrypted. Where IDS10identifies a packet with an encrypted header, IDS10determines that the packet likely originated from an unwanted application, such as Back Orifice, and blocks the packet and future packets of the communication session.

Where IDS10is able to identify the application using a particular communication session, IDS10may either permit or prevent the communication session from continuing. For example, a system administrator may configure IDS10to explicitly allow all identifiable applications, allow all applications except for a specified list of identifiable applications, or prevent all communications except for communications from a specified list of permitted software applications. In cases where IDS10is unable to identify the application, IDS10may further inspect characteristics of the data from the communication session to determine whether the data is encrypted. As one example, IDS10may identify an encrypted communication session by determining whether the data in the communication session exceeds a randomness threshold. A system administrator may configure IDS10to establish or modify the randomness threshold. When the data from the communication session exceeds the randomness threshold, IDS10determines that the data is encrypted. Upon determining that the packet flows of the communication session are encrypted, IDS10applies the principles described herein to determine whether the communication session is associated with an allowed application or a typically unwanted application. For example, IDS10may then determine whether a proper key exchange was initially detected for the communication session and/or whether the communication session contains fully encrypted packets or whether only application-layer data within the packets is encrypted.

IDS10may be configurable in a variety of ways. For example, in one embodiment, IDS10may permit a communication session that is unencrypted to continue, despite being unable to identify a particular application that is using the communication session. In one embodiment, where IDS10determines that a communication session, for which an application cannot be identified, is encrypted, IDS10may terminate the communication session. IDS10may take other actions as well, such as dropping the packet, blocking future sessions from a computing device that initiated the communication session, or other actions. In one embodiment, where IDS10determines that a communication session, for which an application cannot be identified, is encrypted, IDS10may further determine whether a key exchange associated with the communication session can be identified; where a key exchange has been identified, IDS10may permit the communication session and where a key exchange has not been identified, IDS10may terminate the communication session. In one embodiment, IDS10may be configured to take any or all of the above actions, in accordance with a user configuration that a user, such as a system administrator, may configure.

In some embodiments, enterprise network5includes multiple IDSs10and14located within different regions (e.g., sub-networks) of enterprise network5. Security management device18may operate as a central device for managing IDSs10and14. Although the example illustrated inFIG. 1is described in terms of dedicated IDSs10and14, the functionality described herein may be incorporated within other devices, such as firewall9or switch19.

The example embodiment of IDS10portrayed byFIG. 1may provide several advantages. For example, IDS10may discover an undesirable application participating in a network communication session with a network device, such as one of nodes8, of enterprise5. IDS10may also discover the undesirable application as early as the first transmitted packet after a communication session is established, e.g., when the packet is fully encrypted. In this manner, IDS10may block the communication session from the undesirable application, and other traffic of the communication session, as early as the first packet after the communication session is established.

FIG. 2is a block diagram illustrating an example embodiment of an IDS20. In the illustrated example, IDS20includes a forwarding plane22that transparently monitors inbound network traffic24and forwards the network traffic as outbound network traffic26. In the example illustrated byFIG. 2, forwarding plane22includes flow analysis module25, stateful inspection engine28, protocol decoders30, forwarding component31and security management module44.

Security management module44presents a user interface by which administrator42configures IDS20. For example, administrator42may configure IDS20to monitor particular subnets of the enterprise network. In addition, security management module44presents a user interface by which administrator42may specify attack definitions33, which security management module44relays to stateful inspection engine28. In one embodiment, attack definitions33may be compound attack definitions.

Security management module44may further present a user interface by which administrator42may configure IDS20to take particular actions when a fully encrypted packet is identified. For example, the user interface may permit the administrator to drop all fully encrypted packets, log details about the communication session associated with the fully encrypted packets, drop the packets only if a key exchange has not been previously identified for the communication session, throttle down the communication session with fully encrypted packets to reduce the bandwidth usage of that communication session, or other actions. The user interface may also enable administrator42to configure a randomness threshold that IDS20uses to determine whether a packet is encrypted, as described further herein.

Flow analysis module25receives inbound traffic24and identifies network flows within the traffic. Each network flow represents a flow of packets in one direction within the network traffic and is identified by at least a source address, a destination address and a communication protocol. Flow analysis module25may utilize additional information to specify network flows, including source media access control (MAC) address, destination MAC address, source port, and destination port. Other embodiments may use other information to identify network flows, such as IP addresses.

Flow analysis module25maintains data within flow table35that describes each active packet flow present within the network traffic. Flow table35specifies network elements associated with each active packet flow, i.e., low-level information such as source and destination devices and ports associated with the packet flow. In addition, flow table35may identify pairs of packet flows that collectively form a single communication session between a client and server. For example, flow table35may designate communication session as pairs of packet flows in opposite directions for flows sharing at least some common network addresses, ports and protocol.

In one embodiment, to determine whether a packet flow is associated with an identifiable application, stateful inspection engine28buffers a copy of the packet flow and reassembles the buffered packet flow to form application-layer communications. For example, stateful inspection engine28may reconstruct TCP segments into application-layer communications, which represent protocol-specific messages.

Where stateful inspection engine28is unable to determine an identity of an application associated with a packet flow, stateful inspection engine28performs additional analysis on the packet flow. In one embodiment, for example, stateful inspection engine28attempts to determine whether the packet flow is encrypted. When stateful inspection engine28determines that the packet flow is encrypted, stateful inspection engine28further determines whether characteristics of the communication session indicates that the packet flow is associated with an unwanted software application. Stateful inspection engine28may, in one embodiment, determine whether a key exchange has occurred for the communication session when stateful inspection engine28identifies an encrypted packet of the communication session. In this embodiment, when stateful inspection engine28determines that a key exchange has occurred, stateful inspection engine28permits the communication session to continue, but when stateful inspection engine28determines that a key exchange has not occurred, stateful inspection engine28takes a programmed response, e.g., to block the communication session, block future communication sessions from a peer of the communication session, output an alert, or other actions.

In the event a security risk is detected, stateful inspection engine28outputs an alert40to security management module44for logging and further analysis. In addition, stateful inspection engine28may take additional actions, such as dropping the packets associated with the communication session, automatically closing the communication session, or other actions. If no security risk is detected for a given application-layer communication session, stateful inspection engine28instructs forwarding component31to forward the packet flows between the peers. Forwarding component31may, for example, maintain a routing table that stores routes in accordance with a topology of enterprise network5for use in forwarding the packet flows.

FIG. 3is a block diagram that illustrates an example embodiment of stateful inspection engine28. In the example embodiment, stateful inspection engine28includes reassembly module50, application identification module51, and attack detection module52. In addition, stateful inspection engine28includes patterns table54, data buffer55, anomalies table56, and attack definitions33.

Reassembly module50receives inbound network traffic24and reassembles application-layer communications from the packet flows. Reassembly module50forwards the reassembled application-layer communications to the appropriate protocol decoders30for processing.

Stateful inspection engine28stores attack definitions33received from security management module44. Attack definitions33may be stored, for example, in a computer-readable medium, such as random access memory (RAM). Each of attack definitions33specifies a combination of one or more patterns specified within patterns table54and one or more protocol-specific anomalies specified within anomalies table56.

Application identification module51attempts to identify an application associated with each of the intercepted communication sessions. In one embodiment, when stateful inspection engine28receives a packet as part of a packet flow, reassembly module50buffers the packet in data buffer55. Reassembly module50attempts to reconstruct application layer data from the packets in data buffer55. Application identification module51then attempts to identify the application associated with the packets in accordance with this reconstructed data. In other embodiments, application identification module51may use other techniques to attempt to identify the application associated with the communication session.

Application identification module51sends data from the packets to protocol decoders30. When application identification module51is able to determine the application associated with the communication session, application identification module51sends data from the communication session to a corresponding one of protocol decoders30. When application identification module51is not able to identify an application corresponding to the communication session, application identification module51sends the data from the communication session to a default protocol decoder of protocol decoders30.

In any case, protocol decoders30receive data of the communication session from application identification module51and attempt to identify a network protocol being used by the communication session. Protocol decoders30utilize an application-layer header of a packet of the communication session to attempt to identify a protocol being used by the communication session. When the application-layer header is encrypted, however, protocol decoders30may not be able to identify the protocol being used by the communication session. Protocol decoders30represent a set of one or more protocol-specific software modules. Certain ones of protocol decoders30may correspond to a different communication protocol or service. Once protocol decoders30have identified a protocol corresponding to the communication session, protocol decoders30forward the data from the communication session to attack detection module52.

In one embodiment, one of protocol decoders30corresponds to a default protocol decoder. When application identification module51is unable to determine an application corresponding to a communication session, application identification module51sends data from that communication session to the default protocol decoder. The default protocol decoder attempts to identify various elements of the communication session that make the communication session appear to conform to a protocol. For example, the default protocol decoder attempts to analyze the bi-directional communications for the communication session to identify a key exchange for cryptography. As another example, the default protocol decoder attempts to identify regularly occurring patterns that correspond to protocol headers in the packets. Other embodiments may utilize other detection algorithms to determine if the communication session is following a protocol, even if the protocol is unknown.

When the default protocol decoder of protocol decoders30is able to identify that the communication session is following a protocol, the default protocol decoder forwards data from the communication session to attack detection module52. When the default protocol decoder of protocol decoders30is unable to identify a protocol for the communication session, however, the default protocol decoder sends data of the communication session to encryption detection module58.

Encryption detection module58attempts to determine whether the packets of the communication session are fully encrypted, i.e., that the application-layer header as well as the application-layer payload are encrypted. For example, encryption detection module58may extract the payload from a TCP/IP packet, which frequently includes an application-layer packet header (e.g., HTTP header) and application-layer payload. Encryption detection module58then analyzes characteristics of the TCP/IP payload to determine whether the payload of the TCP/IP packet is fully encrypted. That is, encryption detection module58examines the entire payload of a TCP/IP packet, including a region that would typically be an application-layer header, and compare characteristics of the TCP/IP payload to a randomness threshold to determine if the TCP/IP payload that would normally correspond to the application-layer header as well as the application-layer payload are encrypted.

In one embodiment, administrator42(FIG. 2) may configure a randomness threshold of IDS20(FIG. 2). In another embodiment, IDS20utilizes a randomness threshold specified by a randomness detection algorithm, such as the Wald-Wolfowitz runs test algorithm. Encryption detection module58determines a randomness value for a packet of the communication session being examined, i.e., the TCP/IP payload of a packet of the communication session. When the randomness value of a particular TCP/IP payload of the communication session exceeds the randomness threshold, encryption detection module58determines that the communication session is encrypted. Accordingly, encryption detection module58sends a message to attack detection module52that the communication session is being conducted by an undesirable application. Attack detection module52, therefore, may perform one or more of a variety of actions. For example, attack detection module52may terminate the communication session, log the communication session, generate a message for, e.g., administrator42, or take other actions.

Encryption detection module58may implement and execute one or more methods to determine a randomness value. A detailed description of an exemplary one of such algorithms is discussed with respect toFIG. 6. As one example, encryption detection module58may implement the Wald-Wolfowitz runs test. Encryption detection module58may also apply one or more filters to a packet to determine whether the packet is random. Encryption detection module58may use any algorithm that is able to determine a randomness value, and should not be understood to be limited to the Wald-Wolfowitz runs test or filters. Encryption detection module58may also provide a user, such as administrator42, with the ability to apply additional tests for randomness of the data to determine the randomness value, or with the ability to add, remove, or edit the filters.

FIG. 4is a flowchart illustrating an exemplary operation of an IDS in accordance with the techniques described herein. For exemplary purposes, the flowchart is described in reference to IDS20(FIG. 2).

Initially, security management module44receives configuration information from administrator42and, in response, configures IDS20to monitor a network or portions thereof (subnets) of interest (70). During this process, configuration manager44presents a user interface by which administrator42specifies patterns or other attack definitions33. For example, administrator42may configure IDS20to block all packets deemed to be fully encrypted and to log information about the communication session associated with the blocked packets. Administrator42may also configure IDS20to throttle down (i.e., bandwidth limit) the communication session associated with the packets to minimize the bandwidth used by the communication session. Administrator42may also configure IDS20to block future communication sessions from either or both of the communicating devices. IDS20may additionally include a default configuration that specifies actions IDS20should take when IDS20detects a fully encrypted packet. IDS20may, for example, be programmed to let such packets pass by default, or to block all fully encrypted packets by default.

Once configured, IDS20monitors network traffic24(72). In some configurations, stateful inspection engine28of forwarding plane22may receive network traffic and mirror the network traffic for purposes of analysis. Forwarding component31seamlessly forwards the original network traffic.

Stateful inspection engine28of forwarding plane22next receives a packet, e.g. from flow analysis module25(74). While stateful inspection engine28may receive all packets of a particular communication session, for the purposes of this discussion, it is assumed that this packet is the first packet following the packets used to establish the three-way handshake of TCP/IP. In general, IDS20may be capable of identifying certain network attacks as early as with this first packet.

Application identification module51attempts to identify an application associated with the received packet (76). For example, application identification module51may inspect the packet header to determine the destination port of the packet. In some cases, the destination port is associated with the protocol or application being used for the communication session. For example, port80is typically associated with HTTP traffic, while port22is typically associated with SSH traffic. Application identification module51may also inspect an application-layer header, within a TCP/IP payload of the packet, to identify the application associated with the packet. Application identification module51may use any known or future technique to attempt to identify the application associated with the communication session.

When application identification module51is able to determine the application associated with the communication session (“YES” branch of76), application identification module51forwards the packet (78). In one embodiment, application identification module51may send the packet to protocol decoders30to identify the protocol in use for the communication session. In another embodiment, application identification module51sends the packet to attack detection module52to inspect the packet and to determine whether the packet is associated with a network attack. In yet another embodiment, IDS20simply forwards the packet towards its destination in the network.

When application identification module51is unable to determine the application associated with the communication, however, (“NO” branch of76), application identification module51sends the packet to encryption detection module58to determine whether the packet flow represents an encrypted packet flow. In one embodiment, application identification module51may first send the packet to protocol decoders30. Protocol decoders30attempt to determine if the packet resembles a protocol by, for example, looking for a particular pattern of data at particular locations in the packet, such as in the portion of the TCP/IP payload that typically carries an application-layer header.

Encryption detection module58inspects the packet when an application and/or protocol cannot otherwise be associated with the packet and the corresponding communication session. Encryption detection module58determines whether the packet is fully encrypted (80). That is, encryption detection module58examines the packet, including the application-layer header, to determine whether the packet, including the application-layer header, is encrypted. Encryption detection module58analyzes the TCP/IP payload of the packet to determine a randomness value of the TCP/IP payload, which includes the application-layer header as well as the application-layer payload. Where the randomness value of the packet exceeds a threshold, which may be predefined or may be configured by an administrator, encryption detection module58determines that the packet is encrypted.

When encryption detection module58determines that the packet is not fully encrypted (“NO” branch of80), either because the packet is not encrypted at all or because the packet is part of a valid encryption scheme, for example an encryption scheme that does not encrypt the application-layer header, encryption detection module58may forward the packet (78). In one embodiment, IDS20may simply forward the packet toward its network destination. In another embodiment, encryption detection module58may send the packet to attack detection module52to determine whether the packet represents a network attack.

When encryption detection module determines that the packet is fully encrypted, however, (“YES” branch of80), encryption detection module58may output an alert and take a programmed response (82). The response to take may be in accordance with configuration data received from, e.g., administrator42, at step (70). For example, the programmed response may be to terminate the communication session, to block future communication sessions from the peer that initiated the communication session, or other actions.

FIG. 5is a flowchart illustrating another exemplary operation of an IDS in accordance with the techniques described herein. For exemplary purposes, the flowchart is described in reference to another exemplary embodiment of IDS20(FIG. 2).

Initially, security management module44receives configuration information from administrator42and, in response, configures IDS20to monitor a network or portions thereof (subnets) of interest (120). During this process, configuration manager44presents a user interface by which administrator42specifies patterns or other attack definitions33. For example, administrator42may configure IDS20to block all fully encrypted packets and log information about the communication session associated with the packet. Administrator42may also configure IDS20to throttle down the communication session associated with the packet to minimize the bandwidth used by the communication session. Administrator42may also configure IDS20to block future communication sessions from either or both of the communicating devices. IDS20may additionally include a default configuration that specifies actions IDS20should take when IDS20detects a fully encrypted packet. IDS20may, for example, be programmed to let such packets pass by default, or to block all fully encrypted packets by default.

Once configured, IDS20monitors network traffic24(122). In some configurations, stateful inspection engine28of forwarding plane22receives network traffic and mirrors the network traffic for purposes of analysis. Forwarding component31seamlessly forwards the original network traffic.

Stateful inspection engine28of forwarding plane22next receives a packet, e.g., from flow analysis module25(124). While stateful inspection engine28may receive all packets of a communication session, for the purposes of this discussion, it is assumed that this packet is a packet following the packets used to establish the three-way handshake of TCP/IP.

Application identification module51attempts to identify an application associated with the received packet (126). For example, application identification module51inspects the packet header to determine the destination port of the packet. In some cases, the destination port is associated with the protocol or application being used for the communication session. For example, port80is typically associated with HTTP traffic, while port22is typically associated with SSH traffic. Application identification module51may also attempt to identify an application-layer header of the packet and use the application-layer header to identify the application associated with the communication session. Application identification module51may use any known or future technique to attempt to determine the application associated with the communication session.

When application identification module51is able to determine the application associated with the communication session (“YES” branch of126), application identification module51forwards the packet (128). In one embodiment, application identification module51may send the packet to protocol decoders30to identify the protocol in use for the communication session. In another embodiment, application identification module51may send the packet to attack detection module52to inspect the packet and to determine whether the packet is associated with a network attack. In yet another embodiment, IDS20may simply forward the packet towards its destination in the network.

In one embodiment, application identification module51sends the packet to protocol decoders30. Protocol decoders30may attempt to determine if the packet resembles a protocol by, for example, looking for particular data at particular locations in the packet, such as in the application-layer header, and/or by applying pattern matching on portions of the TCP/IP payload. Protocol decoders30may further inspect the packet to determine if the packet is part of a key exchange of the communication session. When protocol decoders30determine that the packet is part of a key exchange of the communication session, protocol decoders30may log this determination in association with the communication session.

As an example, the communication session may be in accordance with the secure socket layer (SSL) protocol. Although application identification module51may not be able to determine the specific application associated with the communication, protocol decoders30may determine that the communication session is using the SSL protocol. The SSL protocol uses several messages to indicate the use of the SSL protocol after the TCP/IP three-way handshake. A client using the SSL protocol will first send a ClientHello message to a server that includes the highest transportation layer security (TLS) protocol version the client supports, a random number, a cipher suite that the client supports, and compression methods supported by the client. The server will respond with a ServerHello message including a selected TLS protocol version, a random number, a cipher suite, and a compression method, each of which the server may select from the client's ClientHello message. Other standard messages are also exchanged between the client and the server to initiate an SSL session. Protocol decoders30may include an SSL decoder that may identify the protocol in use for the communication session as SSL. Moreover, protocol decoders30may recognize a packet that includes key exchange information that is exchanged as part of the SSL protocol. For example, protocol decoders30may recognize a ClientMasterKey message sent by the client to the server. Protocol decoders30may also recognize a ServerVerify message sent by the server to the client. Protocol decoders30may recognize any or all of these messages, for example, by inspecting the application layer packet header.

When application identification module51is unable to determine the application associated with the communication, however, (“NO” branch of126), application identification module51sends the packet to encryption detection module58. Encryption detection module58inspects the packet when application detection module51cannot identify an application associated with the communication session and when protocol decoders30cannot identify a protocol associated with the communication session. Encryption detection module58attempts to determine whether the packet is fully encrypted (128). Encryption detection module58determines a randomness value for the TCP/IP payload of the packet and compares the randomness value to a randomness threshold. Where the randomness value exceeds the randomness threshold, encryption detection module58determines that the packet is fully encrypted.

When encryption detection module58determines that the packet is not fully encrypted (“NO” branch of128), encryption detection module58may forward the packet. For example, in one embodiment, IDS20may simply forward the packet toward its network destination. In another embodiment, encryption detection module58may send the packet to attack detection module52to determine whether the packet represents a network attack.

When encryption detection module determines that the packet is fully encrypted, however, (“YES” branch of128), encryption detection module58may determine whether a key exchange has already been identified for the communication session (130). As discussed above, protocol decoders30may have determined that one or more previously exchanged packet represented a key exchange and logged the fact that a non-hidden (i.e., readily detectable) key exchange has occurred for the communication session. If this has taken place earlier in the communication session, (“YES” branch of130), in this embodiment encryption detection module58determines that the current packet, while potentially fully encrypted, is not associated with an invalid or otherwise unwanted application. Therefore, encryption detection module58may forward the packet. For example, in one embodiment, IDS20may simply forward the packet toward its network destination. In another embodiment, encryption detection module58may send the packet to attack detection module52to determine whether the packet represents a network attack.

However, if there has been no earlier key exchange (“NO” branch of130), encryption detection module may output an alert and take a programmed response (132). The response to take may be in accordance with configuration data received from, e.g., administrator42, at step (120).

Although this embodiment has been described with respect to fully encrypted packets, the techniques may be applied to packets in which only the portions of the TCP/IP payload that typically correspond to the application-layer payloads have been encrypted. For example, such encrypted packets may be discarded when no key exchange for the communication session has been previously detected and logged.

FIG. 6is a flowchart illustrating an exemplary method to determine whether a packet is fully encrypted. For exemplary purposes, the method ofFIG. 6is described with respect to encryption detection module58ofFIG. 3. In the exemplary method depicted inFIG. 6, encryption detection module58may apply the Wald-Wolfowitz runs test to a packet to determine whether the data of the packet is random. In this exemplary method, encryption detection module58may also apply one or more filters to the packet to determine whether the data of the packet is random. In general, any statistical method for determining randomness or entropy of data in packets may be used to determine whether data of a packet is encrypted. That is, encryption detection module58may determine that data of a packet is random by implementing any statistical randomness- or entropy-identifying technique. In general, when encryption detection module58determines that data of a packet is random or has high entropy, e.g., by exceeding a randomness or entropy threshold, encryption detection module58determines that the packet is encrypted.

Another exemplary method for identifying encrypted packets includes tracking two classes for bytes of a packet and comparing the sizes of the classes. One class may represent bytes with values in the hexadecimal range from 0x00 to 0x7F and another class may represent bytes with values from 0x80 to 0xff. Encryption detection module58tracks two counters, one for each of the two classes. When encryption detection module58identifies a byte with a value for the first class, encryption detection module58increments the counter for the first class, and when encryption detection module58identifies a byte with a value for the second class, encryption detection module58increments the counter for the second class. Encryption detection module58then determines that a packet is random, hence encrypted, when the two counters are relatively equal, i.e., within a certain range of each other.

Another exemplary method for determining whether a packet is encrypted includes identifying repeating patterns of varying lengths, e.g., one byte, two bytes, four bytes, or other within a packet. That is, encryption detection module58may determine that a packet is not random when encryption detection module58determines that there exist repeating patterns within the packet.

Initially, encryption detection module58receives a packet (140). For example, encryption detection module58may receive a TCP/IP packet from, e.g., protocol decoders30. Encryption detection module58may then determine the size N of the TCP/IP payload portion or, in another embodiment, just a portion of the TCP/IP payload of the received packet that typically corresponds to an application-layer header (142). In general, the size of the packet's TCP/IP payload refers to the number of bytes in the packet to be analyzed. In one embodiment, encryption detection module58may receive the number of bytes in the packet as an argument to a function or as a configurable input from a user.

Encryption detection module58next determines the mean of the byte values in the packet for the portion of TCP/IP payload being analyzed (144). Encryption detection module58may, for example, add the value of each byte of that portion of the packet in an accumulator variable and divide the total by the total number of bytes N, as determined above, to determine a mean value. Encryption detection module58may then determine a number of “runs” of bytes in the packet, where a “run” is a sequence of bytes that are all above or below the determined mean value (146). For example, for each byte, encryption detection module58may determine whether the value of the byte is greater than or less than the determined mean of the packet. Encryption detection module58may use a data structure, such as an array, to keep track of a comparison of each byte in the packet to the mean value of all bytes in the packet. Where a byte has a value that is greater than the mean, that byte may be assigned a value of “0,” in the array. Where a byte has a value that is less than or equal to the mean, the byte may be assigned a value of “1” in the array. Encryption detection module58may use other data structures, instead of an array, to track the comparison as well.

In general, a “run” is a sequence of adjacent cells in the array that all have the same value. As a simplified example, an array such as “0001111100110000111” has six runs, three “0” runs and three “1” runs. Encryption detection module58determines the total number of runs in the packet by examining each of the cells in the array. For example, encryption detection module58may examine each array cell individually to determine whether that cell constitutes a continuation of a run (i.e., has the same value of the previous cell), or whether that bit constitutes a new run (i.e., has the opposite value of the previous cell). Encryption detection module may also determine the number of runs in other ways.

Encryption detection module58also determines how many of each cell value, i.e. “0” and “1” occur in the packet. Encryption detection module58may make this determination concurrently with determining the number of runs in step (146), or as an additional step. Encryption detection module58may, for example, include two counters, an N0counter to count the number of “0” valued cells and an N1counter to count the number of “1” valued cells. Because the packet is N bytes in length, and each cell will only have the value “0” or “1”, N should equal N0+N1. In the exemplary string above, N is 19, N0is 9, and N1is 10.

Encryption detection module58next determines the expected number of runs in a packet with N bytes (148). In accordance with the Wald-Wolfowitz runs test, the number of runs in a sequence should be a random variable. A random variable has two elements, a mean (μ) and a variance (σ2). Encryption detection module58calculates the mean in accordance with the Wald-Wolfowitz runs test by determining:

μ=2*N0*N1N+1
Encryption detection module58uses the values of N, N0, and N1, determined earlier at steps (142), (144), and (146) to determine the mean. In accordance with the Wald-Wolfowitz runs test, this is the mean number of runs in a packet with N bytes. For the example, μ=10.

Encryption detection module58also determines the variance (σ2) (150). In accordance with the Wald-Wolfowitz runs test, encryption detection module58calculates the variance (σ2) by determining:

σ2=(μ-1)⁢(μ-2)(N-1)
Encryption detection module58uses the value of μ, determined earlier at step (148), to determine the variance. This formula is a simplification of the formula prescribed by Wald-Wolfowitz, which is:

σ2=2*N0*N1*(2*N0*N1-N)N*N*(N-1)
Various embodiments of encryption detection module58may use either the prescribed formula or the simplification thereof from Wald-Wolfowitz for determining the variance (σ2). In accordance with the Wald-Wolfowitz runs test, this is the variance to the mean number of runs in a packet of size N. In one embodiment, administrator42may further configure or refine this equation for determining the variance using security management module44. For the example bit string above, σ2=4.

Encryption detection module58next determines whether the actual number of runs in the packet is within the range predicted by the Wald-Wolfowitz runs test, to determine whether the bits in the packet are random (152). In one embodiment, encryption detection module58determines the randomness of the data in the packet by determining:
|(#_Runs(packet)−μ)/σ|≦x
That is, encryption detection module58compares the absolute value of the difference between the actual number of runs in the packet and the mean number of runs in a packet of size N, as determined by the Wald-Wolfowitz runs test, divided by the square root of the variance. If this value is within a certain range (x), then encryption detection module58determines that the packet is random. Otherwise, encryption detection module58determines that the packet is not random. In one embodiment, a user, such as administrator42, may configure the value of x. The value of x may be, for example, 2.58. In the above example, the number of actual runs is 6, and the mean number of runs is 10, the difference being 4. If the value of x is 2.58, the example sequence “0001111100110000111” could be determined to be random, because the difference (value4) divided by the square-root of the variance (2) is less than 2.58.

When encryption detection module58determines that data in the portion of the packet being tested qualifies as random (“YES” branch of152), encryption detection module58marks the packet as encrypted (158). When, on the other hand, encryption detection module58determines that data in the packet is not random, i.e., below the defined threshold (“NO” branch of152), encryption detection module58determines that the packet is not encrypted (156). In general, a packet from an application following a particular unencrypted protocol has a high amount of runs, which would be outside the number of runs predicted by the Wald-Wolfowitz runs test.

In one embodiment, an administrator, such as administrator42, may add, edit, or remove filters of encryption detection module58. A filter instructs encryption detection module58to determine that a packet is random or not random based on certain filtering characteristics. The filtering characteristics may include, for example, the number of bytes in the packet, examining bytes in the packet for a certain pattern, determining whether a minimum number of byte values above or below the mean occur, or other characteristics of a random or nonrandom packet. The following pseudocode shows one exemplary embodiment of a function performed by encryption detection module58on a packet to determine whether the packet is random or not:

As is evident from the exemplary pseudocode, encryption detection module58may use a combination of the Wald-Wolfowitz runs test and one or more filters to determine whether the contents of a packet are random. Other filters and algorithms can also be used to determine whether a packet is random.

FIGS. 7A-7Bare screenshots illustrating one embodiment of administrator user interface300presented by an exemplary embodiment of security management module44. In general, security management module44presents administrator user interface300that includes two tabs, “Filter” tab302and “Actions” tab304.

FIG. 7Adepicts administrator user interface300when “Filters” tab302is selected, as indicated by highlighting of filters tab302. Filters tab302may be selected either by a user or as a default selection upon initial presentation of administrator user interface300. Filters tab302presents a list of available filters308to a user, such as administrator42. In the example ofFIG. 7A, available filters308include a minimum length for a packet selectable by check box310A, a maximum length for a packet selectable by check box310B, and a filter to determine whether values repeat in the packet selectable by check box310C.

When administrator42, for example, selects check box310A, administrator42may also enter a value into field320. This value may correspond to a minimum length for a packet. When administrator42enters a value into field320and selects save button344, administrator user interface300may instruct security management module44to use a filter to determine whether a packet exceeds a minimum length of the value entered in field320. For example, administrator42may determine that packets that do not exceed a minimum number of bytes are not a concern, so those packets should simply be declared not random.

When administrator42selects check box310B, administrator42may also enter a value into field322. This value may correspond to a maximum length for a packet. When administrator42enters a value into field322and selects save button344, administrator user interface300may instruct security management module44to use a filter to determine whether a packet is less than a maximum length of the value entered in field322. For example, administrator42may determine that packets that exceed a maximum number of bytes are not a concern, so those packets should simply be declared not random.

When administrator42selects check box310A and selects save button344, administrator user interface300instructs security management module44to use a filter to determine whether data in a packet has repeating values. Repeated data in a packet may be an indication that the packet is not random, or that the packet is associated with a protocol that should not be blocked. In one embodiment, encryption detection module58compares data in the packet to other data in the packet to find a repeating sequence. In one embodiment, encryption detection module58stores a sequence of data from each packet in a communication session and determines whether there exists a repeated value in each packet of the communication session; in this case, the repeated value may indicate a standard header of an unknown protocol. In any case, IDS20may utilize a repeating values filter.

Administrator42may also add new filters to security management module44. In the example ofFIG. 7A, administrator42may enter a new filter into field312of administrator user interface300. Upon completing a filter, administrator42may select “add” button314to add the filter to IDS20. Alternatively, administrator42may select “load” button316to retrieve a previously recorded filter from a computer-readable medium, such as a hard drive, a network drive, a flash memory stick, or other location. In this case, administrator42may load the filter and save the filter to IDS20. IDS20applies the new filters added by administrator42, once administrator42selects save button344.

FIG. 7Bdepicts administrator user interface300when “Actions” tab304is selected, as indicated by highlighting of actions tab304. Actions tab304presents detection options panel340and response actions panel360to a user, such as administrator42. In the example ofFIG. 7B, detection options panel340presents two options to administrator42: “Detect fully encrypted packets” and “Detect key exchange.” Administrator42may select either option using a check box, such as one of check box342A or check box342B. Other embodiments may use other selection methods, such as radio buttons, drop down lists, or other selection methods. In one embodiment, when check box342is not selected, other options of administrator user interface300, such as check boxes343A-343D and the corresponding text within response actions panel360, may be displayed in a grayed-out format, and administrator user interface300may prevent a user from selecting check boxes343A-343D.

When administrator42selects “Detect fully encrypted packets” check box342A, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Detect fully encrypted packets” check box342A is selected, administrator user interface300sends a message to security management module44to detect fully encrypted packets, or to operate in a detect fully encrypted packets mode. Accordingly, security management module44instructs stateful inspection engine28to analyze incoming packets to determine whether those packets are fully encrypted. Stateful inspection engine28then begins to determine whether incoming packets are fully encrypted, for example, using the methods disclosed herein, or other methods for detecting encrypted packets.

When administrator42selects “Detect key exchange” check box342B, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Detect key exchange” check box342A is selected, administrator user interface300sends a message to security management module44to detect key exchanges for monitored communication sessions, or to operate in a detect key exchange mode. Accordingly, security management module44instructs stateful inspection engine28to analyze incoming packets to determine whether those packets represent a key exchange for their corresponding communication session. Stateful inspection engine28then begins to determine whether incoming packets represent a key exchange as part of the communication session. In one embodiment, stateful inspection engine28may only determine whether a first set of packets of a communication session represents a key exchange. In one embodiment, stateful inspection engine28may only determine whether packets represent a key exchange when those packets correspond to a communication session for which an application and/or a protocol cannot be identified.

In one embodiment, administrator42may select both “Detect fully encrypted packets” check box342A and “Detect key exchange” check box342B. In other embodiments, administrative user interface300may limit administrator42to selecting exactly one of “Detect fully encrypted packets” check box342A and “Detect key exchange” check box342B. In still other embodiments administrative user interface300may limit administrator42to selecting only one of “Detect fully encrypted packets” check box342A and “Detect key exchange” check box342B or neither of the two.

In the example ofFIG. 7B, check box342A is checked, as marked with an “X” within check box342A, and check box342B is unchecked, without the presence of an “X” in check box342B. Other embodiments may use other visual cues to indicate whether a check box is checked. For example, in one embodiment, check boxes342A-342F (check boxes342) may indicate a “checked” status by including a check mark in a checked one of check boxes342.

Actions tab304of administrator user interface300also presents response actions panel360to a user, such as administrator42. Response actions panel360presents four options to administrator42: “Drop packets”, “Throttle session”, “Send alert message”, and “Log detection.” Administrator42may select these options using check boxes342C-342F, respectively. Other embodiments may include other responses, such as blocking future communication sessions from a computing device that sent a fully encrypted packet, or blocking the communication session in which a fully encrypted packet was identified.

When administrator42selects drop packets check box342C, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Drop packets” check box342C is selected, administrator user interface300sends a message to security management module44to drop packets associated with the communication session corresponding to options selected in detection options panel340. For example, when “Detect fully encrypted packets” check box342A is selected, security management module44instructs stateful inspection engine28to drop packets associated with a communication session in which a packet that was fully encrypted was detected. As another example, when “Detect key exchange” check box342B is selected, security management module44instructs stateful inspection engine28to drop packets associated with a communication session when those packets are fully encrypted but no identifiable key exchange has been detected.

When administrator42selects throttle session check box342D, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Throttle session” check box342E is selected, administrator user interface300sends a message to security management module44to throttle any communication session corresponding to options selected in detection options panel340. For example, when “Detect fully encrypted packets” check box342A is selected, security management module44instructs stateful inspection engine28to throttle a communication session in which fully encrypted packets were detected. As another example, when “Detect key exchange” check box342B is selected, security management module44instructs stateful inspection engine28to throttle communication sessions in which a fully encrypted packet is detected without first detecting a key exchange for that communication session.

When administrator42selects send alert message check box342E, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Send alert message” check box342E is selected, administrator user interface300sends a message to security management module44to send an alert message to, e.g., administrator42. Accordingly, when IDS20detects a fully encrypted packet, security management module44sends an alert to administrator42. The alert may be any sort of an alert, e.g., a pop-up window, an e-mail, a text message, a page, an audible alert, or any other sort of an alert to inform administrator42that IDS20has detected, e.g., a fully encrypted packet.

When administrator42selects log detection check box342F, and then selects save button344, administrator user interface300sends a corresponding message to security management module44. In particular, when “Log detection” check box342A is selected, administrator user interface300sends a message to security management module44to make a log of the event that was identified. For example, security management module44records the source port, destination port, source IP address, destination IP address, time of discovery, packet size, a copy of the packet, other actions taken in response to the detection, or any other information that administrator42may find useful. Security management module44may create this log in, e.g., a database, a text file, or any appropriate data structure.

When administrator42selects cancel button346, in one embodiment, check boxes342revert to a previous setting, e.g. the setting as of the last time save button344was selected. In one embodiment, when administrator42selects cancel button346, check boxes342revert to a default setting. When administrator42selects cancel button346, administrator user interface300sends a message to security management module44to perform in accordance with a setting associated with a selection of cancel button346. In one embodiment, security management module344closes administrative user interface300, i.e. remove the administrative user interface300window from view on, e.g., a computer monitor displaying administrative user interface300.

Methods described herein may be performed in hardware, software, or any combination thereof within a network device. For example, methods described herein may be performed by an application specific integrated circuit (ASIC) or a general-purpose processor. Methods described herein may also be embodied in a computer readable medium containing instructions. Instructions embedded in a computer readable medium may cause a programmable processor, or other processor, to perform the method, e.g. when the instructions are executed. A computer readable medium may be a computer readable storage medium. Computer readable storage media may include, for example, random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media.