Patent ID: 12206697

DETAILED DESCRIPTION

It should be noted that while the following description is drawn to fabric networking nodes, some operating as a fabric manager, various alternative configurations are also deemed suitable and may employ various computing devices including routers, switches, interfaces, systems, databases, engines, controllers, or other types of computing devices operating individually or collectively. One should appreciate the computing devices comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, solid state drive, RAM, flash, ROM, etc.). The software instructions preferably configure the computing device to provide the roles, responsibilities, or other functionality as discussed below with respect to the disclose apparatus. In especially preferred embodiments, the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on TCP, UDP, IP, Ethernet, wireless signally, HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods. Data exchanges preferably are conducted over a packet-switched network, the Internet, LAN, WAN, VPN, or other type of packet switched network.

One should appreciate that the disclosed techniques provide many advantageous technical effects including a fabric communication infrastructure capable of self-detecting anomalous behavior among distinct physical nodes within the fabric at fine level of granularity.

As used herein, and unless the context dictates otherwise, the term “coupled to” is intended to include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements). Therefore, the terms “coupled to” and “coupled with” are used synonymously.

InFIG.1, network fabric100comprises multiple networking nodes130and provides a communication infrastructure among internal devices110and external remote devices160over Internet155. Fabric100can exist as part of a corporate network spread over a large geographic region, government network spread across the planet, or other type of network. In some embodiments, the network can be integral with the Internet. As illustrated network fabric100is considered an internal fabric which is under control of a single entity (e.g., organization, company, etc.). Devices110are considered internal to fabric100because they operate under authority granted by the entity while devices160are external from fabric100because they operate outside the jurisdiction of the entity.

Each node130preferably comprises a networking switch operating as a hybrid-fabric apparatus cable of transporting data across fabric100from one networking node130to another while also providing a distributed application engine. Distributed applications can be deployed on nodes130and executed as software instructions. Each node130can include processors, memory, ports, or other apparatus components that can be individually assigned to data transport operations, an application's execution, or other role or responsibility. One example of a suitable switch that can be adapted for use with the disclosed techniques includes the Raptor Networks Technology™, Inc. ER-1010 switch. Another example of suitable hybrid-fabric network apparatus is described in co-pending U.S. patent application having Ser. No. 12/849,521 to Wittenschlaeger titled “Hybrid Transport—Application Network Fabric Apparatus”, filed Aug. 3, 2010.

Each node130can be fungible with respect to the other nodes130, and can be configured to operate as a fabric manager capable of managing one or more aspects of fabric100. Nodes130can also measure one or more behavior metrics representative of a behavior of fabric100. As fabric100operates, nodes130measure or collect behavior metrics. A fabric manager can compare aggregated metrics with nominal behaviors. Furthermore, the metrics can be stored in a black box memory recording a history of the fabric from the node's local perspective. The history, metrics, behaviors or other data can be captured at a fabric level, apparatus level, or even down to a component level (e.g., processor, memory port, etc.).

InFIG.2, networking node230represents a hybrid-fabric apparatus and has been outfitted with black box memory232. The memory itself can include a non-volatile memory (e.g., ROM, Flash, SSD, disk, etc.) preferably secured against tampering. In some embodiments, black box memory232can be configured to comply with FIPS140requirements. Black box memory232can store behavior metrics, vectors of behavior metrics of interest, historical data, or other information. If networking node230becomes damaged, black box memory232can provide some information regarding events leading to the failure of node230.

In some embodiments, network node230includes anomaly agent234. Anomaly agent234can be configured to acquire or manage measured vectors of behavior metrics, or interact with a remote fabric manager. As desired agent234can calculate a status from the metrics with respect to anomaly detection criteria, or anomaly criterion, preferably as a function of a variation from a baseline vector of behavior metrics. Furthermore, agent234can communicate a calculated status with other nodes230via one or more data channels within the fabric. In some embodiments, a channel can be dedicated for distributing anomaly statuses among nodes. For example, a fabric manager can established that all anomaly status traffic should be exchanged over a specific wavelength (e.g., a logical channel) on optic fiber connections among nodes. Such an approach ensures that anomaly status information does not interfere with general data transport.

FIG.3provides an overview of a method300for detecting an anomalous behavior within a fabric based on one or more measured vectors of behavior metrics as compared to vectors representing nominal behavior.

At step310, a nominal behavior can be characterized as a baseline vector. The vector can comprise behavior metrics related to the fabric where the behavior metrics can be associated with internal aspects of the fabric or external elements (e.g., remote devices, remote addresses, etc.) beyond the edge of the fabric, as least to the extent visible to the node measuring the metrics. Behavior metrics can represent aspects of the operating fabric according to various levels of granularity. For example metrics can range from fabric-level metrics (e.g., total aggregated load, bandwidth, etc.), apparatus-level metric (e.g., load on a node, total data exchanged through node, available channels, etc.), or even down to component-level metrics (e.g., available memory, CPU load, etc.). Metrics can be single valued (e.g., available memory, latency, etc.) or multi-valued (e.g., array of values with time-stamps). Metrics can also be considered dynamic reflecting different values at different times, or can be composites of multiple basic metric. An example composite metric might include a calculated value derived from current load value divided by number of active ports. Such a metric could be considered a load/port density.

One should appreciate that the baseline vector is more than a mere list of metrics having values. The vector can include a representation of how the metrics behave with time, how the metrics correlate with each other, or reflect other properties representing a dynamic behavior of a fabric. In some embodiments, the baseline vector is derived by monitoring the fabric over a period of time and establishing the baseline vector by looking for correlated metrics. For example, multi-variate analysis can be performed with respect to metrics to determine if one metric is correlated with another. Other algorithms could also be used to establish baseline vectors or anomalous conditions including genetic algorithms, neural networks, bloom filters, or other known AI techniques. Vectors can be considered a manageable data object that can be stored, moved, updated, removed, or otherwise managed as an independent object from other objects. One aspect of the inventive subject matter is considered to include instantiating a baseline vector object from one or more generic vector objects.

Each member of a vector can be constructed to represent various aspects of the fabric including the fabric as a whole, an apparatus, or a component as desired. Still further, a vector can also comprise member elements that reflect non-fabric elements possibly including remote devices, remote addresses, weather conditions, sensor data, geo-locations, news events, sporting events, stock prices, or other attributes associated with non-fabric information. Vectors can also comprise other vectors as a member element.

Baseline vectors can take on many different forms. A baseline vector can represent an overall nominal behavior of a fabric as well as different types of nominal behavior. For example, multiple baseline vectors can be established to represent aspects of the fabric. One could establish a baseline vector representing nominal traffic flow through the fabric (e.g., routes, data loads, latency, etc.) while other vectors could represent operating health (e.g., temperature, error rates, etc.).

Step320includes establishing anomaly detection criteria as a function of a variation from the baseline vector where the criteria represent an aspect of a possible anomalous behavior (seeFIG.4). The criteria can be constructed to represent various conditions reflecting an anomalous behavior. The detection criteria can reflect that (a) an anomalous behavior has occurred, (b) an anomalous behavior is about to occur, or (c) an anomalous behavior is likely to occur. One should note the criteria depend on a variation from the baseline vector as opposed to solely based on deviations of one or more metrics. The variation can be calculated based on a variation function applied to measured behavior metric having the same member elements as the baseline vector. One example variation function can include a Chi-Square fit to members of the measure vector compared to the baseline vector. If the Chi-Squire value exceeds a threshold, an anomalous behavior might be present. It should be appreciate that temporal properties can be incorporated in the measurement of the variation. All vector-based variation functions are contemplated.

A single set of anomaly detection criteria can be associated with a type of anomaly (e.g., intrusion, attack, error condition, maintenance problem, etc.) or even with a specific anomaly (e.g., denial of service attack). The anomaly detection criteria can be stored as an object and can include attributes describing the types of anomaly for which the detection criteria is applicable. As discussed further below, a single set of anomaly detection criteria can comprise indicators for multiple, distinct anomaly types.

The anomaly detection criteria can be established using numerous techniques. In some embodiments, a fabric manager can access a human manager interface to define the anomaly detection criteria. Other more preferable and automatic methods are also contemplated. For example, step323can include simulating an anomalous behavior within the fabric while the fabric is active. Running a simulation is considered to include running a self-contained software application designed to mimic the fabric experiencing anomalous behavior. In some embodiments, where the networking nodes of the fabric include hybrid-fabric apparatus capable of general purpose computation, the simulation can be deployed on one or more of the networking nodes. Such an approach allows for gaining an understanding of how an anomalous behavior affects vectors of behavior metrics without substantially impacting the fabric in general. Real-world, real-time existing metrics can be fed into the simulation to gain a better simulation. Running a simulation of an anomalous behavior is considered advantageous because the simulation can be run multiple times to build statistics reflecting variations around the baseline vector or reflecting anomalous behavior. However, simulations likely produce less accurate results than a real-world experience.

Another technique for establishing anomaly detection criteria can include step325. Step325includes modeling an anomalous behavior within the fabric while the fabric is operating. Modeling the behavior is considered to include constructing a logical representation of the fabric by using actual nodes that exchange data among each other within boundaries of a baseline vector. An anomalous behavior can be introduced or injected into the modeled fabric, and then the effects can be directly observed in a more accurate real-world environment. A vector of behavior metrics can be measured and compared to the baseline vector to determine how the measured behavior vector varies from the baseline vector. Thus, the variation can be measured directly from real-world or real-time conditions. Modeling the fabric is contemplated to provide more accurate results over a simulation. However, a model of the fabric can adversely impact fabric performance. Furthermore, it would likely be difficult to build statistics around the modeling of the various behaviors. It is also contemplated that both simulating and modeling the fabric can be used to derive anomaly detection criteria.

In addition to modeling the fabric, step327can include running a live drill. Consider a government fabric that requires high security. The managers of the fabric can conduct a live drill by introducing an anomalous behavior on a live, active fabric. The managers are essentially modeling the behavior while directly observing or collecting data reflecting measured vectors of behavior metrics.

Step330can include disaggregating the anomaly detection criteria into one or more anomaly criterion. Each criterion is considered to be function of a measured vector of behavior metrics rather than merely a metric threshold value. The function can result in a continuous set of values representing a status of the criterion. An anomaly criterion, by itself, would likely fail to provide sufficient insight into an anomalous behavior. However, when the statuses of most, if not all, of the anomaly criterion are aggregated, sufficient information might be available to indicate that an anomalous behavior is present.

Step340includes disseminating anomaly criterion to one or more networking nodes within the fabric. Each node can be responsible for comparing locally obtained behavior metrics forming a measured vector of behavior metrics against the node's stored anomaly criterion. An anomaly criterion is also considered to be dynamic criterion that can change with time. For example, the criterion can include a behavior profile of one or more measured vector of behavior metrics. Furthermore, the criterion can be updated if required due to the fabric environment changing.

Step343can include updating the anomaly detection criteria currently being monitored by the fabric. Updating the detection criteria can include changing the anomaly detection criteria, and in response, the anomaly criterion stored on the nodes. Updating the conditions for detecting an anomalous behavior allows for taking into account the fabric's changing state while still being considered to fall within a nominal range of behavior. For example, a baseline vector could present a steady state traffic flow where no applications are executing within the fabric. When a new application is introduced to the fabric, the anomaly detection criteria can be changed to ensure that an anomaly detection event is not erroneously triggered. If the anomaly detection criteria are updated, the anomaly criterion can also be updated if desired, but might not necessary because the updated detection criteria might still be able to leveraged existing disseminated criterion.

Step350includes having each node calculate a criterion status for each criterion for which it is responsible. As mentioned previously, each criterion can comprise a function, or algorithm, applied to measured vector of behavior metrics. The result of the function, single valued or multi-valued, represents a status. The status can include a binary satisfied or not-satisfied indicator or can include other values as well including a real number, a continuous value, a string, or other types of data. When desired, the status can be sent by a node's anomaly agent to a remote node for further analysis. Statuses can be sent periodically, in response to a polling event, continuously, or with other frequency. By sending only a status as opposed to raw data, the amount of traffic on the network is reduced and the impact on general data transport is minimized. Thus, the fabric can be self-monitored in a distributed, parallel manner.

Step353can include networking nodes collecting or otherwise obtaining vectors of behavior metrics that can be analyzed in real-time relative to the anomaly criterion present at the node. Obtaining the behavior metrics can include measuring metrics locally, calculating behavior metrics, reading sensors, accessing the memory or black box for historical data, exchanging behavior metrics or criterion status with other nodes, or any other type of collecting data. Collecting metrics can include collecting fabric-level metrics, apparatus-level metrics, component-level metrics, or even metrics associated with one or more applications running within fabric.

As indicated by step355, calculating a criterion status can include calculating the status based on a trend in measured behavior metrics. For example, a variation from the baseline vector could include a variation in a rate of change of a time varying vector, metric, or rate of change of one metric relative to another (seeFIG.4). A trend can be established by accessing historical vectors or metric values, possibly stored in the black box memory, and analyzing the metrics for observed trends. One should appreciate that a trend is considered to be a patterned behavior of one metric relative to another. Typically a trend is with respect to time. However, a trend can be with respect to other metrics as well as mentioned above.

Step360includes aggregating anomaly criterion statuses from at least some of the networking nodes that monitor their own anomaly criterion. Aggregating the statuses can include obtaining the statuses over a management channel in the fabric; a logical or physical channel. Typically a network node operating as a fabric manager with respect to anomaly detection aggregates the statuses. Aggregating the statuses can include polling for the statuses, listening for statuses, obtaining statuses as part of a heartbeat, or other form of receiving statuses. In some embodiments, the statuses can be sent to a remote device, either internal to the fabric or external to the fabric, as desired.

Step370includes detecting satisfaction of the anomaly detection criteria as a function of the anomaly criterion statuses, where satisfaction appears to indicate an anomalous behavior is present. Although satisfaction of the anomaly detection criteria can be a binary result (i.e., satisfied or not-satisfied), in more preferred embodiments satisfaction can vary by degrees according the structure of the anomaly detection criteria. Furthermore, the statuses are not required to indicate satisfaction of a condition. For example, anomaly detection criteria can be partially satisfied where some of the criterion statuses collected from network nodes might indicate the criterion itself has not yet been satisfied or might reflect that a correlated metric outside the bounds of the baseline vector has changed.

Allowing for the anomaly detection criteria to be partially satisfied lays a foundation for a number of opportunities. As suggested by step373, partial satisfaction can give rise to generating a leading indicator of an anomalous behavior. As criterion statuses are aggregated, the anomaly detection criteria can become “more” satisfied, or possibly less satisfied. At step375based on the level of satisfaction of the anomaly criteria or the collected criterion status, one can generate a likelihood of the anomalous behavior is about to occur. The likelihood can represent a probability that the anomalous behavior might occur. In some embodiments, the likelihood can be calculated based on compiled statistic reflecting nominal behavior relative to historical anomalous behavior (e.g., compiled from modeling or simulating anomalous behaviors). Such information can be useful to fabric managers so the managers can address possible risks or take pre-emptive actions.

Step377can include identifying an anomaly type of a detected anomalous behavior based on the aggregated anomaly criterion statuses. The statuses, in aggregate, can provide an indication of an anomalous behavior signature. One should note that the signature is based on the aggregated statuses rather than merely a collection of metrics. As stated before, a status does not necessarily equate to a metric. The fabric manager can consult one or more databases, possibly located in the manager's memory, to identify a signature. In some embodiments, a status can be used as an index into an anomaly database to find anomaly types.

Example anomaly types can include attacks, intrusions, infiltrations, natural behavior, unexpected behavior, random events, or other types of anomalous behaviors. As indicated anomaly types do not necessarily have to relate to a fabric threat, but could be a result of natural causes. For example, a natural behavior could result in an unusually high, but natural load that might cause detection of an anomalous behavior. An unexpected behavior could include a cluster of minor events that push a vector of measured behavior metrics outside the detection window of a baseline vector. A random event could simply include a break in a communication line. All anomaly types are contemplated.

Anomalous behaviors can also be classified as being internal to the fabric or external to the fabric. Internal anomalous behaviors reflect an anomalous behavior of the fabric itself or its constituents. One should also appreciate that an anomalous behavior can be detected with respect to external devices communicatively coupled with the fabric. As external devices, or other entities, interact with the fabric, the external device could exhibit unusual behavior beyond an expected nominal behavior where the unusual behavior can be detected based on the disclosed techniques. In view of such a capability, the fabric can distinguish between internal threats originating internally and external threats originating external to the fabric.

Step380can include notifying a manager of the anomalous behavior. The manager can be a node operation as a fabric manager, a human being, or other entity having responsibility for managing the fabric. Notifications can be sent via an electronic communication channel (e.g., SMS, email, network management application, SNMP, etc.) as desired.

A notification of the anomalous behavior can include one or more instructions on how to respond to the anomalous behavior. In some embodiments, step383can include migrating anomalous traffic to a monitored data channel. For example, if an intrusion is detection, the fabric can automatically reconfigure a routing topology used by the intruder so that the intruder's packets are routed toward a network operations center, a data sink, a honey pot, or the location so the anomalous traffic can be further analyzed in real time.

When the anomaly type is known, as suggested by step385, the notification could also include instructions on what actions should be taken to respond to the anomalous behavior based on anomaly type. Actions can include storing historical data within the black box memory, migrating black box data to another node's black box, reconfiguring a routing topology, locking one or more data channels or connected devices, or other actions.

FIG.4is presented for further clarity and represents a graphical representation400of two correlated metrics in a baseline vector. The metrics have a moderately strong correlation as shown where a first metric monotonically increases with respect to the second metric. The two metrics can compose a baseline vector, which can also include a derived relationship between the two metrics (e.g., a trend). Through observation of a nominal behavior represented by correlation410, anomaly detection criteria420are derived as a function of variation430around correlation410. Detection criteria420are illustrated as an envelope around correlation410. The envelope could include contours represent degrees of detection where criteria420can be partially satisfied. When a current behavior varies away from the baseline, as indicated by the variation arrow, detection criteria420can be checked. If variation430satisfies the anomaly criteria then an anomalous behavior is considered to (a) have occurred, (b) is occurring, or (c) is about to occur according the conditions set forth in the criteria.

One should note, as illustrated there can be a single anomaly detection criteria (i.e., the envelope). However, a variation from the baseline vector could be attributed to many differ types of anomalies. Although the graph is presented in two dimensions: as M1versus M2, it is also contemplated that correlations or criteria could be established based on any number of metrics so that the criteria, variation, correlations, vectors, or other objects could have higher dimensions beyond just two.

It should be apparent to those skilled in the art that many more modifications besides those already described are possible without departing from the inventive concepts herein. The inventive subject matter, therefore, is not to be restricted except in the scope of the appended claims. Moreover, in interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms “comprises” and “comprising” should be interpreted as referring to elements, components, or steps in a non-exclusive manner, indicating that the referenced elements, components, or steps may be present, or utilized, or combined with other elements, components, or steps that are not expressly referenced. Where the specification claims refers to at least one of something selected from the group consisting of A, B, C . . . and N, the text should be interpreted as requiring only one element from the group, not A plus N, or B plus N, etc.